Model and method for evaluation of eQNet “travel well” quality learning resources Assoc. Prof....

Model and method for evaluation of eQNet “travel well” quality learning resources

Assoc. Prof. Dr. Eugenijus Kurilovas, ITC, LithuaniaEdReNe final strategic seminar, Barcelona, 26 March 2010

• LRE particularly seeks to identify LOs that “travel well” (i.e., reusable) across national borders and can be used in a cultural and linguistic context different from the one in which they were created

• eQNet is a three-year (September 2009-2012) Comenius Multilateral Network funded under the European Commission’s Lifelong Learning programme. The project is coordinated by European Schoolnet and involves 9 Ministries of Education - AT, BE, CZ, IT, LT, NO, PT, SK, SE or agencies nominated to act of their behalf

• The primary aim is to improve the quality of LOs in LRE. eQNet will do this by establishing a network consisting of policy makers, researchers, and practitioners (teachers) that will develop and apply “travel well” quality criteria to both existing LRE content as well as that to be selected in future from national repositories

eQNet project

• The vision driving the LRE is that a significant percentage of high quality LOs developed in different countries, in different languages and to meet the needs of different curricula can be re-used at European level

• eQNet will provide a forum for joint reflection and co-operation related to the exchange and re-use of educational content and allow network members to:

1. better share information and expertise particularly related to “travel well” quality criteria (pedagogical, technical and intellectual property rights (IPR) factors)

2. develop new frameworks (including a network of expert teachers) to improve the quality of LOs and metadata in both national repositories and the LRE, including the growing volume of user-generated content and metadata, as well as to improve the multilinguality of LRE content as a result of the translation of metadata, making use, where appropriate, of automatic metadata translation approaches and technologies

3. enable schools to participate in a community of practice related to the use LOs at European level

eQNet project

Major results will include:

• the development of “travel well” quality criteria to more easily identify LOs with the potential for cross-border use (this WP is coordinated by the Lithuanian MoE represented by ITC, and EUN)

• the practical application by teachers of these criteria to >3,500 LOs in the LRE

• ‘showcases’ of the best of these LOs in a “travel well” section of the LRE portal

• where necessary, the enrichment of selected LOs with new or better metadata

• a community of practice for teachers around these LOs

eQNet project

• The aim of the presentation is to suggest scientific model and method for expert evaluation of quality of LOs paying especial attention to LOs reusability level

• The main problem of all existing approaches in the area is high level of expert evaluation subjectivity

• Several scientific methods and principles to minimise subjectivity level in expert evaluation of LOs quality were applied, namely:

1. principles of multiple criteria decision analysis for identification of quality criteria

2. technological quality criteria classification principle3. fuzzy group decision making theory to obtain evaluation

measures4. normalisation requirement for criteria weights5. scalarisation method for LOs quality optimisation

• The complex application of these approaches could significantly improve the quality of expert evaluation of LOs and noticeably reduce the expert evaluation subjectivity level

• An example of practical application of these approaches for LOs quality evaluation is also presented

Applied scientific methods and principles

• To present different approaches to evaluation of Learning Objects (LOs) – previous works

• To present the principles of identification of quality evaluation criteria

• To present LOs technological quality evaluation model (criteria tree)

• To present overall LOs quality evaluation model (incl. technological, pedagogical and IPR criteria)

LOs quality evaluation model (criteria tree) – Outline

The main notions:

• LO is referred here as “any digital resource that can be reused to support learning” (Wiley, 2000)

• Quality evaluation is defined as “the systematic examination of the extent to which an entity (part, product, service or organisation) is capable of meeting specified requirements” (ISO/IEC 14598-1:1999, 1999)

• Reusability of LOs (or their ability to “travel well” between different contexts and education systems) is considered here as a part of the overall quality of LOs

• This means that any high quality LO has some reusability level (or potential to “travel well”), but this does not mean that any reusable LO is quality one

LOs quality evaluation model (criteria tree)

The need for reusability of LOs has at least three elements:

• Interoperability: LO is interoperable and can be used in different platforms

• Flexibility in terms of pedagogic situations: LO can fit into a variety of pedagogic situations

• Modifiability to suit a particular teacher’s or student’s needs: LO can be made more appropriate to a pedagogic situation by modifying it to suit a particular teacher’s or student’s needs (McCormick et al., 2004)


Learning Object Review Instrument (LORI):

• Presentation: Aesthetics• Presentation: Design for learning• Accuracy of content• Support for learning goals • Motivation• Interaction: Usability• Interaction: Feedback and adaptation• Reusability• Metadata and interoperability compliance• Accessibility


Leacock & Nesbit (2007) quality criteria:

• Content quality: veracity, accuracy, balanced presentation of ideas, and appropriate level of detail)

• Learning goal alignment: alignment among learning goals, activities, assessments, and learner characteristics

• Feedback and adaptation: adaptive content or feedback driven by differential learner input or learner modelling

• Motivation: ability to motivate and interest an identified population of learners

• Presentation design: design of visual and auditory information for enhanced learning and efficient mental processing

• Interaction usability: ease of navigation, predictability of the user interface, and the quality of the interface help features

• Accessibility: design of controls and presentation formats to accommodate disabled and mobile learners

• Reusability: ability to use in varying learning contexts and with learners from different backgrounds

• Standards compliance: Adherence to international standards and specifications


Becta (2007) core pedagogic principles:

• Inclusion and access• Learner engagement• Effective learning• Assessment to support learning• Robust summative assessment• Innovative approaches• Ease of use• Match to the curriculum


Becta (2007) core design principles:

• Digital learning resource design• Robustness and support• Human-computer interaction• Quality of assets• Accessibility• Interoperability• Testing and verification• Effective communication


Paulsson and Naeve (2006) have suggested six areas for establishing LO technological quality criteria:

• A narrow definition• A mapping taxonomy • More extensive standards• Best practice for use of existing standards • Architecture models• The separation of pedagogy from the

supporting technology of LOs


MELT set of the LOs quality criteria has been divided into five categories:

• Pedagogical• Usability• Reusability• Accessibility• Production


MELT: the cross-border reuse of content will be more likely if LOs:

• Have a strong visual element and users can broadly understand what is the intended learning objective or topic. E.g., LOs may have little or no text; and include animations and simulations that are self-explanatory or have just a few text labels or icons / buttons for start, stop, etc.

• Have been designed to be language customisable and are already offered in more than one language

• Address curriculum topics that could be considered trans-national

• Are adaptable from a technical (e.g., LOs are supplied along with an authoring environment or tools) or IPR perspective


LOs Q4R (Quality for Reuse) criteria:

• Strategies before LO inclusion in the LOR are based on the following principles:

• Only build or integrate LOs, which can be certified for quality• Interactive LOs are software and as such should answer to

software quality criteria

• Strategies during LO inclusion in the LOR are based on:• The principle of reducing form-filling• Use of guiding wizards, smart automatic and semi-automatic

computer agents to assist in assuring technical interoperability

• Strategies after LO inclusion in the LOR are based on the following principles:

• Provide interesting and easily understood user statistics, such as stars, percentages, voting systems

• Include recommendations for reuse by the user, both to the next user and the designer


Belgium (Flanders) nationally recognised LOs quality criteria:

• Scope Criteria: • Aims and users (Aims; Target group)• Contents (Curriculum area; Relevant source; Source

type; Granularity; No advertisements)• Geographical coverage (Geographical Limitations

(availability – relevance); Language)• Accessibility (Technology; Costs; Registration)• Source data (Metadata)

• Quality criteria:• Content criteria (Validity; Accuracy; Authority;

Actuality; Substantial Coverage; Completeness)• Formal criteria (Navigation; User support; Use of ICT

and standards)• Process criteria (Integrity of information; Site

stability; Platform stability)


Czech LOs quality criteria:

• Content accuracy

• Connection with national curriculum

• Didactics / Pedagogy accuracy

• Typographic rules

• Citations, Copyright


French LOs quality criteria:

• Curriculum conforming

• Free rights for pedagogical use in the classroom and out of the classroom

• Respect of publishers engagements

• Provide access to expertise the resources

• Update information must be available

• Provide the possibility to block advertising (if included)


Lithuanian LOs quality criteria:

• Methodical aspects• User interface (incl. personalisation)• LOs arrangement possibilities • Communication & collaboration

possibilities and tools • Technical features (incl. working

stability)• Documentation • Implementation and maintenance

expenditure


Norwegian LOs quality criteria:

• Technical interoperability (File formats, standards (IMS QTI & CP, etc.))

• Metadata tagging

• Accessibility (WCAG 1.0)

• Parallel publishing (two forms of the Norwegian language exist)

• Pedagogical issues


Portuguese LOs quality criteria:

• Technical

• Scientific

• Educational

• Language

• Values and Attitudes


Swedish LOs quality criteria:

• Every web site must have source information and must not conflict with Swedish law

• The presentation must be clear and easy to navigate

• The information must be reliable: who is responsible for the information, why the information is published, how often it is updated, etc.

• Other criteria deal with general interest, language and access requirements


Swiss LOs quality criteria:

• National level:• Recommendation for description

(Application profile)• Technical tools• Harvesting

• Local / Partner level:• Selection and validation of resources

according local criteria and culture• Metadata check, enrichment and

maintenance• Publication (local)


What is important:

• Alternatives (LOs to be chosen)

• Principles of identification of quality evaluation criteria

• Stakeholders


• The main problem here is how to establish the ‘proper’ set of LOs evaluation criteria which should reflect objective scientific principles of construction a model (criteria tree) for LOs quality evaluation

• These general principles have been analysed in several multiple criteria decision analysis (MCDA) related works such as (Belton, Stewart, 2002)

• If the set of decision alternatives is assumed to be predefined, fixed and finite, then the decision problem is to choose the optimal alternative or, maybe, to rank them. But usually the experts (decision makers) have to deal with the problem of optimal decision in the multiple criteria situation where the objectives are often conflicting. In this case, an optimal decision is the one that maximises the decision maker’s utility.

• An initial candidate set of key factors, which we term criteria should emerge from the problem structuring process

• In identifying these, the following considerations are relevant to all multiple criteria decision analysis approaches:


• Value relevance: Are the decision makers able to link the concept to their goals, thereby enabling them to specify preferences which relate directly to the concept?

• Understandability: It is important that decision makers have a shared understanding of concepts to be used in an analysis.

• Measurability: All MCDA implies some degree of measurement of the performance of alternatives against specified criteria, thus it must be possible to specify this in a consistent manner. It is usual to decompose criteria to a level of detail which allows this.

• Non-redundancy: Is there more than one criterion measuring the same factor? When eliciting ideas often the same concept may arise under different headings… One can easily check for criteria which appear to be measuring the same thing by calculating a correlation coefficient if appropriate data is available, or carrying out a process of matching as associated with analysis of repertory grids.


• Judgmental independence: Criteria are not judgementally independent if preferences with respect to a single criterion, or trade-offs between two criteria, depend on the level of another.

• Balancing completeness and conciseness: a number of authors note that desirable characteristics of a value tree are that it is complete, i.e., that all important aspects of the problem are captured, and also that it is concise, keeping the level of detail to the minimum required.

• Operationality: The model is usable with reasonable effort – that the information required does not place excessive demands on the decision makers. The context in which the model is being used is clearly important in judging the usability of a model.

• Simplicity versus complexity: The value tree, or criteria set is itself a simple representation, capturing the essence of a problem, which has been extracted from a complex problem description. The modeller should strive for the simplest tree which adequately captures the problem for the decision maker.


LOs quality evaluation model (technical criteria tree)

Software engineering Principle: we can evaluate the software using the two types of evaluation criteria – ‘internal quality’ and ‘quality in use’ criteria:

• ‘Internal quality’ is a descriptive characteristic that describes the quality of software independently from any particular context of its use (decision makers – scientists, experts in informatics engineering)

• ‘Quality in use’ is evaluative characteristic of software obtained by making a judgment based on criteria that determine the worthiness of software for a particular project or user (decision makers – users)


EUN has identified some “travel well” parameters:

• Language independence / Low language dependence (easily translatable) or Multilinguality

• Strong visual structure (animations, images and short videos are travelling best)

• No large semantic density (easy to include in any curriculum)

Other parameters more relating to quality in general:

• Cultural appropriateness: the resource respects the cultural and social norms of teacher and / or pupil

• Usability: users can find their way through the resource almost intuitively. This may be possible even if the user is unfamiliar with the language of the resource

• Technical features: easy to access (no login or plug-ins)

• Openness: open license, free to use, open code (like in flash resources) (Panzavolta, 2009)


• To present the ratings (values) of LOs evaluation criteria (taking into account technological, pedagogical and intellectual property rights (IPR) criteria)

• To present the method for evaluation of LOs (experts additive utility function)

• To provide an example of practical evaluation of three LOs alternatives using this method

LOs quality evaluation method – Outline

• Expert evaluation is referred here as the multiple criteria evaluation of the learning software (such as LOs) aimed at selection of the best alternative based on score-ranking results.

• Learning software multiple criteria evaluation method is referred here as the experts additive utility function including the alternatives’ (i.e., different LOs) evaluation criteria, their ratings (values) and weights.

• This method is well-known in the theory of optimisation methods and is named “scalarization method”.

LOs quality evaluation method


m

iii XfaXf

1

)()(

11

m

iia 0ia

whereX – LO alternative

i = {1, … , m} – evaluation criteria (m=8) – rating (value) of criterion

– ‘normalized’ weights of criteria

0ia

,

,

)(Xfi

The widely used measurement criteria of the decision attributes’ quality are mainly qualitative and subjective. Decisions in this context are often expressed in natural language, and evaluators are unable to assign exact numerical values to the different criteria. Assessment can be often performed by linguistic variables: ‘bad’, ‘poor’, ‘fair’, ‘good’ and ‘excellent’. These values are imprecise and uncertain: they are commonly called fuzzy values. Integrating these different judgments to obtain a final evaluation is not evident.

Therefore, (Ounaies et al., 2009) have proposed to use fuzzy group decision making theory to obtain final assessment measures.

Linguistic variables conversion into non-fuzzy values:Excellent 0.850Good 0.675Fair 0.500Poor 0.325Bad 0.150



Value ExplanationBad (0.150)

Failed or feature (criterion) does not exist

Poor (0.325)

Has poor support and / or it can be done but with significant effort

Fair (0.500)

Fair support but needs modification to reach the desired level of support

Good (0.675)

Good support and needs a minimal amount of effort

Excellent (0.850)

Excellent support and meets the criteria out of the box, minimal effort


Examples of Lithuanian LOs (Physics):

LO1: Light diffraction (Šviesos difrakcija)http://mkp.emokykla.lt/imo/lt/mo/330/

LO2: Photo effect (Fotoefektas)http://mkp.emokykla.lt/imo/lt/mo/367/

LO3: Isobar process (Izobarinis procesas)http://mkp.emokykla.lt/imo/lt/mo/395/

http://mkp.emokykla.lt/imo/lt/mo/330/

LOs quality evaluation method (equal weights 0.125) Evaluation criteria LO1 LO2 LO3

Technological quality criteria

Technological reusability 0.850 0.850 0.850

Design and usability 0.850 0.850 0.850

Working stability 0.850 0.850 0.850

Architecture 0.675 0.675 0.675

Pedagogical quality criteria

Multimedia interactivity level 0.500 0.500 0.500

Language independence 0.850 0.675 0.500

Ease of use, intuitiveness 0.850 0.850 0.850

IPR criterion

Open licence, cost 0.850 0.850 0.850

Evaluation results (values * 0.125)

0.7844 0.7625 0.7406

LOs quality evaluation method (weights 0.2; 0.1)

Evaluation criteria LO1 LO2 LO3

Technological quality criteria

Technological reusability (weight 0.2)

0.1700 0.1700 0.1700

Design and usability (weight 0.1) 0.0850 0.0850 0.0850

Working stability (weight 0.1) 0.0850 0.0850 0.0850

Architecture (weight 0.1) 0.0675 0.0675 0.0675

Pedagogical quality criteria

Multimedia interactivity level (weight 0.1)

0.0500 0.0500 0.0500

Language independence (weight 0.2)

0.1700 0.1350 0.1000

Ease of use, intuitiveness (weight 0.1)

0.0850 0.0850 0.0850

IPR criterion

Open licence, cost (weight 0.1) 0.0850 0.0850 0.0850

Evaluation results 0.7975 0.7625 0.7275

• Table 1 results mean that LO1 meets 78.44% quality in comparison with the ideal, LO2 – 76.25%, and LO3 – 74.06%. According to these experimental evaluation results, LO1 is the best alternative (among the evaluated) from overall quality point of view.

• Table 2 results mean that LO1 meets 79.75% “travel well” quality in comparison with the ideal, LO2 – 76.25%, and LO3 – 72.75%. According to these experimental evaluation results, LO1 is the best alternative (among the evaluated) from “travel well” quality point of view.


• ISO/IEC 14598-1:1999 (1999). Information Technology – Software Product Evaluation – Part 1: General Overview. First edition, 1999-04-15

• Becta (2007). Quality principles for digital learning resources. Available online at http://schools.becta.org.uk/index.php?section=re&catcode=ss_res_dig_02&rid=13293

• Kurilovas, E.; Dagienė, V. (2009). Multiple Criteria Comparative Evaluation of e-Learning Systems and Components. Informatica. 2009, Vol. 20, No. 4, pp. 499-518 [ISI Web of Science]

• Kurilovas, E.; Dagienė, V. (2009). Quality Evaluation and Optimisation of e-Learning System Components. In: Proceedings of the 8th European Conference on e-Learning (ECEL’09). Bari, Italy, October 29–30, 2009, pp. 315–324 [ISI Web of Knowledge] [ERIC] – the best paper award

• Kurilovas, E., Dagiene, V. (2008). Learning Objects and Virtual Learning Environments Technical Evaluation Tools. In: Proceedings of the 7th European Conference on e-Learning (ECEL’08). Agia Napa, Cyprus, 6–7 November 2008, Vol. 2, pp. 24–33 [ISI Web of Knowledge] [ERIC] – the best paper award

References

• Leacock, T. L. & Nesbit, J. C. (2007). A Framework for Evaluating the Quality of Multimedia Learning Resources. Educational Technology & Society, 10 (2), pp. 44–59

• McCormick, R., Scrimshaw, P., Li, N., and Clifford, C. (2004). CELEBRATE Evaluation report. http://celebrate.eun.org/eun.org2/eun/Include_to_content/celebrate/file/Deliverable7_2EvaluationReport02Dec04.pdf

• MELT (2008). EU eContentplus programme’s MELT (Metadata Ecology for Learning and Teaching) project web site. Available: http://melt-project.eun.org

• Panzavolta, S. (2009). Travel-well resources. Presentation in the EMINENT conference. Vilnius, 26-27 December, 2009

• Paulsson, F., Naeve, A. (2006). Establishing technical quality criteria for Learning Objects. Available: http://www.frepa.org/wp/wp-content/files/Paulsson-Establ-Tech-Qual_finalv1.pdf

References

• Ounaies, H.Z.; Jamoussi, Y.; Ben Ghezala, H.H. (2009). Evaluation framework based on fuzzy measured method in adaptive learning system. Themes in Science and Technology Education. Vol. 1, Nr. 1, 2009, pp. 49–58

• Q4R (2008). Quality for Reuse (Q4R) project web site. Available: http://www.q4r.org

• Van Assche, F., and Vuorikari, R. (2006). A Framework for Quality of Learning Resources. In: European Handbook for Quality, CEDEFOP, 2006

• Vargo, J., Nesbit, J. C., Belfer, K., Archambault, A. (2003). Learning object evaluation: Computer mediated collaboration and inter–rater reliability. International Journal of Computers and Applications, Vol. 25 (3), pp 198–205

• Wiley. D. A. (2000). Connecting Learning Objects to Instructional design Theory: a definition, a Metaphor, and a Taxonomy. Utah State University. http://www.reusability.org/read/

References

Thank you for your attention !!!

Questions ?

Dr. Eugenijus [email protected]

mailto:[email protected]

mailto:[email protected]

Model and method for evaluation of eQNet “travel well” quality learning resources Assoc. Prof....

Documents

Transcript of Model and method for evaluation of eQNet “travel well” quality learning resources Assoc. Prof....