ALOA conceptual model: Aligning Learning Outcomes and Assessment
-
Upload
rita-falcao-de-berredo -
Category
Education
-
view
904 -
download
0
Transcript of ALOA conceptual model: Aligning Learning Outcomes and Assessment
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-23
Evaluation of the Application of E-learning Methodologies to the
Education of EngineeringRita Falcão de Berredo
Programa Doutoral em Media Digitais
Especialidade Criação de Audiovisual e de Conteúdos InterativosUniversidade do Porto :: FEUP
Porto, 21st of January of 2013
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Main concepts
ASSESSMENTLEARNINGOUTCOMES
What do we hope students will learn?
How do we know that they have learned?
What should be assessed?
ALIGNMENT
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Before…
ASSESSMENT
LEARNINGOBJECTIVES
What will we teach our students?
ALIGNMENT
CONTENT
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• Demand for new skills and competences• Mobility and recognition• Quality approach and accreditation
Why change?...
THE FOCUS IS ON THE STUDENTS!
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• Students may have prior knowledge and skills• Students must learn what they need for the
profession and future• Provide evidence of acquired competences
And now what?
How can we help teachers select an adequate assessment strategy for the learning outcomes?
ASSESSMENTLEARNINGOUTCOMES
ALIGNMENT
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• Students• Faculty• Course and curriculum developers• Internal quality and accreditation• Mobility and recognition• Employers• General public
To whom is this relevant?
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Why this theme?
European project VIRQUAL
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Learning outcomes are statements of what a learner is expected to know, understand and/or be able to demonstrate after completion of learning. (AHELO - Assessment of Higher Education Learning Outcomes by OECD)
Learning Outcomes
A common language, building blocks, genetic code
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Assessment: Any procedure used to estimate student learning for whatever purpose. (Brown et al)
Assessment
e-Assessment is the use of ICT and the Internet in particular for the assessment of learning, including design, delivery and/or recording of responses. (JISC)
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Alignment
Alignment: The level of correspondence between objectives, instruction and assessment.
(Anderson et al)
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Initial Research
LEARNINGOUTCOMES
Where can I find LOs in Engineering Education?
Courses Programs Qualification Frameworks
ASSESSMENTALIGNMENT
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• Four qualification frameworks– ABET: Accreditation Board for engineering and
Technology – EUR-ACE: European quality label for engineering
degree programmes– CDIO: Conceiving, Designing, Implementing,
Operating– EQF: European Qualification Framework
Learning Outcomes in Engineering
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Initial Research
ASSESSMENT Which types of assessment should be considered?
peer
formative
group
essay
project
chat
MCQ
exam
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• Multiple choice questions • Short Answer Questions • Problems• Essays• Practical work• Reflective practice
Assessment
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Alignment
Specific
Assessment tasks
Course and Programme
LOs
General
General assessment
methods
LOs QFs
ALIGNMENT
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Alignment scenariosAt course level…
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Simple problem…?
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
The ALOA conceptual modelAligning Learning Outcomes and Assessment
ALIGNMENT
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
The rBloom Matrix
Adapted from Anderson et al
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
The ALOA conceptual modelAligning Learning Outcomes and assessment
ALIGNMENT
rBLOOMMATRIX
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• Write LOs statements
• Define content
• Define learning activities
• Define assessment
• Ensure alignment
Teacher with NEW course
ALOAHELP!!
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Analysis of LOsThe student should be able to describe the main
components of a personal computer.
NOUN / KNOWLEDGE
VERB / COGNITIVE PROCESS
rBloomMatrix
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Compare matrixes
ALOADB
ASSESSEMENT 1
ASSESSEMENT 2
ASSESSEMENT 3
ALIGNEDASSESSMENT STRATEGY
MATRIXES OF LOs of course
ALOA SYSTEM
LOsMATRIXESLOs
MATRIXESLOsMATRIXES
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• LOs statements written
• Content defined
• Existing learning activities
• Assessment strategy defined
• Verify alignment
Teacher with EXISTING course
ALOA
HELP!!
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
ALOADB
ASSESSEMENT 1
ASSESSEMENT 2
ASSESSEMENT 3
ASSESSMENT METHODS(EXISTING)
MATRIXES OF LOs of course
ALOA SYSTEM
ALIGNMENT REPORT
e.g.: Accreditation purposes
Compare matrixes
LOsMATRIXESLOs
MATRIXESLOsMATRIXES
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Work overview• Four case studies• LOs and assessment methods• Application of the ALOA• Improvement of ALOA
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• The ALOA conceptual model• A structured approach to assessment• Linking Bloom and assessment• Bloom as a translation for LOs• Clarification of the alignment concept
Major contributions of this work to the field of study
Reflection Quality Mobility
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Future work
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-23
Thank you.
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• Describing– Analyzing and describing the LOs – Analyzing and describing assessment methods
• Aligning– Aligning assessment and LOs
• Mapping– LOs of different levels (programme/course/QF) – LOs of different sources (recognition)– Assessment
Roles of the rBloom Matrix
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• In terms of knowledge it is considered that the design process includes mainly procedural and conceptual.
• The cognitive tasks will include planning and producing an application according to criteria. Some interpretation of criteria will be necessary as well as iterative evaluation of the application against the given requisites.
Step 1: Analysis of LOsStudents should be able to design and implement a simple distributed application.
KNOWLEDGE
Procedural Conceptual
COGNITIVE PROCESS
Understand: interpretingApply: implementingEvaluate: checking, critiquingCreate: planning, producing
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
STEP 2: Build rBloom matrixKNOWLEDGE
Procedural Conceptual
COGNITIVE PROCESS
Understand: interpretingApply: implementingEvaluate: checking, critiquingCreate: planning, producing
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Scenarios of applicationApplication and stakeholders
Proposal of internal alignment
Single courseTeachers, curriculum developers
Verification of internal alignment
Single courseTeachers, curriculum developers, accreditation
Vertical alignment Course/programme/QFcurriculum developers, accreditation
Horizontal alignment Several courses, prior learningTeachers, recognition and mobility
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• RQ1) Which type of Learning Outcomes in the field of Engineering are relevant and should be considered? – Defining and describing LOs in engineering
• RQ2) Which are the e-assessment methods that should be considered? – Defining and describing assessment methods and
e-assessment tasks
Research questions
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• RQ3) What type of intended Learning Outcomes can be measured by assessment methods? – Defining a model of achieving alignment between
learning outcomes and assessment• RQ4) Is it possible to propose specific
assessment strategies for each type of LO in EE?– Applying the model to link specific learning
outcomes with specific assessment tasks
Research questions
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Alignment criteriaCriteriaMatch LO statement = TLA = assessment
Emphasis LOs have different levels of importance
Coverage The content of the course should be covered
Precision A complex LO may include different competences with different values.
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Backwash effect (Biggs)Perspective of the teacher
Perspective of the student
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• Difficulties of applying the model• Testing the templates • Validity of selected list of assessment • Validity of classification and description of the
assessment methods • Validity of selected QFs• Validity of tool
Case studied
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Assessment cycle
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
Validity and reliability
Precision Accuracy
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• Remember• Understand• Apply• Analyse• Evaluate• Create
MCQ
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• speculative• quote to discuss• assertion,• write on• describe/explain• discuss• compare• evaluate• problem
Essays
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• Simple closed ended• Complex closed ended• Open ended
• Routines• Diagnosis• Strategy• Interpretation• Generation
Problems
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• Select crucial evidence• Explain methods, procedures and relationships• Present arguments• Describe limitations of data• Formulate valid conclusions• Identify assumptions• Formulate hypothesis• Formulate action plans
SAQs
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
• Concrete experience • Reflective observation• Abstract Conceptualisation • Active Experimentation•
Reflective practice
Click to edit Master title style
• Click to edit Master text styles– Second level• Third level
– Fourth level» Fifth level
03-05-2303-05-23
MCQ Computer based test / online testing, Optical reading, CAT: Computerized Adaptive Testing
Essays File Upload, Essay Question in online exam, Discussion ForumPublished media, Wiki, Concept maps, Videoconferencing, Chat
Problem solving Computer based test / online testing, File upload, Chat, Concept maps and Diagrams, Simulation, Scenario-based activity
Practicals File upload, Computer based tests/Online testing, Video file, animations and sequence of images, Videoconferencing, Diagrams Publish media or wiki, Chat and discussion forum, Virtual Labs and Remote Labs, Simulation, Scenario-based activity, Game based learning
SAQs Computer based test / online testing, Chat or Forum, Concept maps and Diagrams
Reflective Practice
Portfolios