Outcomes Based EducationPlanning & Management

What, How, WhoLueny Morell, Founder & Director InnovaHiEd

Former IFEES President, ABET Program Evaluatorlueny.morell@gmail.com

www.luenymorell.comwww.innovahied.com

Agenda for today

• The need – why we need to reshape our curricula based on learning outcomes

• The how – outline of the major steps to develop curricula based on outcomes

• Reflection

We are already in the 4th Industrial Revolution

Humans as Robots

Humans as Innovators

https://agenda.weforum.org/2015/09/fourth-industrial-revolution/

“Our future competitiveness

depends on a skilled workforce of

engineers and innovators.”

~Dr. Ray O. Johnson, SVP & CTO, Lockheed Martin

Engineering is much more than technology knowledge

Engineering competencies employers seek (2015)• leadership • the ability to work in a team

structure.• written communication skills• problem-solving skills• strong work ethic• analytical/quantitative skills

http://naceweb.com/about-us/press/class-2015-skills-qualities-employers-want.aspx

Clayton Christensen Harvard University

“In the spirit of honoring traditions, universities hang on to past practices imperiling their future.”

We are teaching 21st

century students with 20th century curricula in 18th century classrooms

I have a cum-laude degree in molecular biometrics from

Cambridge; a PhD in quantum physics from MIT; gold medal in….

Yes… but what can you DO?

Definitions

Learning Outcomes

• Statements that specify what learners will know or be able to do as a result of a learning activity.

• Outcomes are usually expressed as knowledge, skills, or attitudes.• Focus on learner’s behavior that is to be changed.• Serve as guidelines for content, instruction, and evaluation.• Identify specifically what should be learned.• Convey to learners exactly what is to be accomplished and how it will

be measured (assessed).

Edited from http://www.aallnet.org/Archived/Education-and-Events/cpe/outcomes.html

Relationship between objective and outcome

Objective = Aim Outcome = Result

Competencies

Observable behaviors that encompass the knowledge, skills, and personal characteristicsthat distinguish levels of performance in the work.

http://www.aallnet.org/Archived/Education-and-Events/cpe/outcomes.html

Other categories of outcomes (ABET)

• Technical• the specialized skills that are required by a practitioner in the discipline.

• Business• the skills required to function within a larger enterprise.

• Communication• the skills to convey information effectively using a variety of methods and media.

• Professionalism• the personal and professional conduct and qualities expected for a practicing

engineer. • Individual

• skills such as creativity, leadership, innovation, and practical ingenuity are desirable qualities that can be emphasized to the degree that meets a program’s mission.

According to new proposed criteria for ABET (2015)

outcomes = competencies

Σ knowledge, skills and attitudes

Why use learning outcomes/objectives in higher education?• They are important in the teaching/learning process so that BOTH

teachers and students understand the purpose of the interchange.• Having an organized set of objectives helps teachers to:

• plan and deliver appropriate instruction• design valid assessment tasks and strategies • ensure that instruction and assessment are aligned with the

objectives.

http://discoverlibrary.vanderbilt.edu/primo_library/libweb/action/dlSearch.do?institution=VAN&vid=VANDERBILT&onCampus=true&indx=1&bulkSize=10&search_scope=Unicorn&query=any,contains,0321084055&tab=acorn

Question

Which category of outcomes is more important?Knowledge,Skills, or,Attitudes?

It’s not what you know…

but what you can do with what you know!

IN ENGINEERING -SKILLS/ATTITUDES & VALUES ARE CRITICAL

Attributes employers seek in engineers

77.8% - leadership and the ability to work in a team structure> 70% - written communication skills, problem-solving skills, strong work ethic

http://naceweb.com/about-us/press/class-2015-skills-qualities-employers-want.aspx

ABET Outcomes (2015-16)a. an ability to apply knowledge of mathematics, science, and engineeringb. an ability to design and conduct experiments, as well as to analyze and interpret datac. an ability to design a system, component, or process to meet desired needs within realistic constraints

such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability

d. an ability to function on multidisciplinary teamse. an ability to identify, formulate, and solve engineering problemsf. an understanding of professional and ethical responsibilityg. an ability to communicate effectivelyh. the broad education necessary to understand the impact of engineering solutions in a global, economic,

environmental, and societal contexti. a recognition of the need for, and an ability to engage in life-long learningj. a knowledge of contemporary issuesk. an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.

www.abet.org

Professional skills of the 21st century technology leaders

• Mobability — ability to work in large groups; talent for organizing & collaborating with many people simultaneously

• Influency — ability to be persuasive in multiple social contexts & media spaces; understanding that each context & space requires a different persuasive strategy & technique

• Protovation – fearless innovation in rapid, iterative circles

• Emergensight — ability to prepare for & handle surprising results & complexity

• Cooperation Radar — the ability to sense, almost intuitively, who would make the best collaborators on a particular task

• ….. Adapted from Dr. Bob Johansen, President and CEO of the Institute for the Future*Daniel Goleman

The 5 skills of disruptive innovators

• Associating• Connecting the dots, visualizing problem & solution

• Questioning• Break out of the status quo; consider new

possibilities• Observing

• Detect small behavioral details that suggest new way of doing things

• Networking• Gain radically different perspectives by connecting

with people from diverse backgrounds• Experimenting

• Relentlessly try on new experiences

How do innovators come up with new ideas?

Skills to solve the Engineering Grand Challenges1. Practical experience2. Up-to-date knowledge3. Knowledge of fundamentals (e.g.,

calculus)4. Knowledge of theory5. Knowledge of multiple engineering

disciplines6. Understanding of non-engineering

disciplines7. Creativity8. Ability to take risks9. Technical prowess10. Technical intuition11. Responsibility and ethics12. Effective communicator13. Ability to apply digital technologies14. Multi-cultural sensitivity15. Ability to work in a team

Source: 2015 SPEED Global Survey

Other important work skills in 2020• Sense making• Social intelligence• Novel & adaptive thinking• Cross cultural competency• Computational thinking• New media literacy• Transdiciplinary• Design mindset• Cognitive load management• Virtual collaboration

cdn.theatlantic.com/static/front/docs/sponsored/phoenix/future_work_skills_202.pdf

A NEW BREED OF ENGINEERDiverse, interdisciplinary, & all flavors of creative.

A locally relevant, globally competent engineer

HOW CAN WE DESIGNAN OUTCOMES-BASED

CURRICULUM?

Outcomes-based curriculum: ∑multiple learning activities that help achieve competencies

1Course

Course

workshop

Experience

etc

2Course

Course

Course

Experience

Course

etc

3Course

Course

Course

Course

Internship

Research

4Course

Course

Course

Internship

Capstone Design

Students Engineeringcompetencies

APPLY THEENGINEERING

METHOD!

5 Steps1. Define and validate your program

outcomes/competencies2. Plan (MAP) the learning process3. Measure & analyze learning outcomes4. Share & discuss results5. Make decisions and re-engineer to improve

1 - Define and validate your program outcomes/competencies

• Benchmark• Accreditation bodies• Industry • Industry advisory boards• Faculty• Students• Alumni• Professional societies• …

Writing learning outcomes

• They should clearly indicate the nature and/or level of learning required to achieve them successfully.

• They should be achievable and assessable and use language that learners (and other teachers) can easily understand.

• They relate to explicit statements of achievement and always contain verbs.

• Objectives should be SMART: Specific, Measurable, Achievable, Realistic and Timebound.

Bloo

m’s

Taxo

nom

y of

Lea

rnin

g

“At the end of the semester (or, program, learning experience) students will be able to•Understand….•Apply….•Work in teams•Make oral presentations•…”

Examples

At the end of this program the student will demonstrate life-long learning skills

At the end of this course the student will understand professional & ethical responsibilities

Once you develop your list of outcomes you need to validate it with constituents, especially industry & other employers

Think-Pare-Share Exercise

• List your program’s learning outcomes/competencies (knowledge + skills + values).

• Compare to the lists shared. Is anything missing?

ABET Outcomes (2015-16)a. an ability to apply knowledge of mathematics, science, and engineeringb. an ability to design and conduct experiments, as well as to analyze and interpret datac. an ability to design a system, component, or process to meet desired needs within realistic constraints

such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability

d. an ability to function on multidisciplinary teamse. an ability to identify, formulate, and solve engineering problemsf. an understanding of professional and ethical responsibilityg. an ability to communicate effectivelyh. the broad education necessary to understand the impact of engineering solutions in a global, economic,

environmental, and societal contexti. a recognition of the need for, and an ability to engage in life-long learningj. a knowledge of contemporary issuesk. an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.

www.abet.org

5 Steps1. Define and validate your program outcomes/competencies2. Plan (MAP) the learning process3. Measure & analyze learning outcomes4. Share & discuss results5. Make decisions and re-engineer to improve

2 – Plan the learning process to have students develop competencies1. Matrix (map) of

competencies across the curriculum (what, when, depth of learning)

2. Teaching/learning strategies, spaces, resources, etc. that will catalyze the development of competencies

roadmapping

Learning Outcomes Outcome 1 - Evaluate and use big data systems engineering to analyze, evaluate, and design technologies in an enterprise setting

Outcome 2 - Design solutions to big data challenges taking economic and societal constraints into account

Outcome 3 - Work in multi-disciplinary, multi-stakeholder, culturally diverse teams

Outcome 4 - Communicate effectively with multiple stakeholders

Outcome 5 - Leverage and influence professional networks

Outcome 6 - Apply security and ethical standards

Outcome 7 - Manage projects and time effectively

Outcome 8 - Respect and embrace diversity and cultures

Outcome 9 - Be flexible and adaptive

Courses/Depth of Learning*

CSBD 5001: Introduction to Big Data Technologies

3 - 1 1 1 1 - 1 1

CSBD 5050: Big Data Leadership and Entrepreneurism

3 1 3 3 3 3 3 3 3

CSBD 5002: Big Data Infrastructure

5 5 1 1 1 3 1 1 1

CSBD 5003: The Art and Science of Inquiry

5 5 1 1 1 3 1 1 1

CSBD 5004: Big Data Visualization and Analytics

5 5 1 1 1 3 1 1 1

CSBD 5020: Big Data Governance and Stewardship

5 5 1 1 1 5 1 1 1

CSBD 5008: Special Topics in Big Data

3 to 5 3 to 5 1 1 1 3 1 1 1

CSBD 5009: Big Data Focus Elective

3 to 5 3 to 5 1 1 1 3 1 1 1

CSBD 5010: Big Data Industry Practicum

5 5 1 1 1 3 1 1 1

CSBD 5051: Big Data Capstone Project

5 5 1 1 1 5 1 1 1

depth of learning

the Informatics Eng. degree at ETSINF – UPV

Eduardo Vendrell, Dean of Informatics @ UPV

What teaching/learning experiences we need to develop outcomes?

How do people learn? At the brain level, learning occurs when…

Practicewhen learners practice something, the "neurons that control and drive that action fire repeatedly”.

Emotionthe better learning feels, or the more positively learners associate good feelings with the learning process, the more likely they are to be motivated and engaged.

Zull, J. 2004

44

Learning Styles and Learning

• Everybody has a preferred learning style

• We teach the way we learn

• We (professors) expect students to be like us!

• Everybody loses!

Students learn more when they are intensely involved and they are working in real life problems

“Collaborating with others in solving problems… prepares students to deal with the messy, unscripted problems they will encounter daily, both during and after college.”

-US National Survey on Student Engagement, 2010

Examples of teaching/learning strategies that can help develop outcomes

Competency Examples of Teaching/Learning Strategies(a) an ability to apply knowledge of mathematics, science, and engineering

• Lectures• Laboratories• Tinker

(b) an ability to design and conduct experiments, as well as to analyze and interpret data

• Laboratories (non-recipe)• In-class demonstrations• Tinker• Benchmark solutions

(c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability

• Project/problem based learning in courses or throughout the curriculum• Senior Capstone Design• Design centric program• Global engineering challenges• Benchmark solutons

(d) an ability to function on multidisciplinary teams • Working in teams (classroom, homework, projects, etc.)• Working in teams with other disciplines• Organize them in ‘mini-companies’ with roles & responsibilities

(e) an ability to identify, formulate, and solve engineering problems • Have them look for problems in everyday life. Have them formulate them and describe solution’s constraints

• Give them incomplete or ill-defined problems• Benchmark solutions

(f) an understanding of professional and ethical responsibility • Formal course (ethic in the engineering profession)• Share ethics case studies• Ethics debate

(g) an ability to communicate effectively • Write and present problem/projects solutions, proposals, designs, interim reports…

• for diverse audiences and…• using various media (oral, ppt, word, video, internet, etc)• Suggest or have them take technical writing courses/seminars/workshops

(h) the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context

• Bring news and journal items to the class and discuss/debate them• Have them research a topic related to the class• Ask questions during class relating the topic to global, economic,

environmental & societal impact• Field trips• Experts in the classroom• Benchmark solutions

(i) a recognition of the need for, and an ability to engage in life-long learning

• Ask them to summarize a reading, analyzing critically what’s read• Ask them to learn a topic through internet (MOOC, Khan, etc.)• Promote participation in professional societies (e.g., IEEE)• Experts in the classroom• Teach someone else• Promote graduate school or studying other disciplines

(j) a knowledge of contemporary issues • Problems, projects, design to address current needs (local, global)• Have them summarize and discuss current issues in class• Experts in the classroom• Benchmark solutions

(k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.

• Use modern tools and techniques in class, labs, projects, etc. • Experts in the classroom• Benchmark solutions

Use active learning!

48

The benefits of active/experiential learning have been proven

Ref. Interactive-engagement vs traditional methods: A six-thousand student survey of mechanics test data for introductory physics courses by Richard R. Hake, American Journal of Physics, Vol 66, pp 64-74, 1998

Interactive EngagementHeads-on and hands-on activities

Traditional MethodsPassive lectures, recipe labs, and algorithmic problem exams

Active learning approaches

• Writing, presenting• Debating ethics cases• Cooperative/collaborative learning• Practical experiences

• Industry internships, real projects/problems, visits, invited experts

• Demonstrations, labs• Project/problem based learning (PBL)• CDIO• Flipped classroom• Blended classroom• … Anything but just lecture!

5 Steps1. Define and validate your program outcomes/competencies2. Plan (MAP) the learning process3. Measure & analyze learning outcomes4. Share & discuss results5. Make decisions and re-engineer to improve

How do we know students have learned desired outcomes?

3 - Assessment cycles (internal, external)

Internal Cycle –Student Learning

2-Implement

3-Measure

Outcomes

4- AnalyzeMake

Decisions5- Share

1-Learning

Objectives

Aligned with institution’s mission,~3 years,

with stakeholders input

External Cycle –

Learning Objectives

2-Measure

Outcomes

3- AnalyzeMake

Decisions

1- Learning Objectives

Continuous (don’t have to assess everything all the time!)

Examples of assessment planning matrices 1- tools vs outcomes 2 – tools vs time, who

ASSESSMENT STRATEGIES MATRIX Utilization Strategy – Timing – Responsibility

Assessment Tools

Pre-Engineering Freshman Orientation Questionnaire at UNIV-0004 Freshman Orientation Course (by Departmental Counselor) Ethics Integration Assessment Form at UNIV-0004 Freshman Orientation Course (by Departmental Counselor)

Course Assessment Laboratory Reports (copies of) at all Laboratory Courses (by Lab Instructors) Exams, Quizzes, Homeworks (copies of) retain examples of these tools (by all Professors/Instructors) EIT Exam Statistical Report obtain annually from Examining Board (by Department) Written Report Evaluation anytime written reports are required (labs, etc) (by all Professors/Instructors) Oral Presentation Assessment at all student oral presentations (by all Professors/Instructors) Teamwork Assessment Form (I) at end of any semester where work done in groups (Professors/Instructors) Teamwork Assessment Form (II) at end of any semester where work done in groups (Professors/Instructors) Peer Evaluation Form at end of any semester where work done in groups (Professors/Instructors) Undergrad Research Exp Assessm. Form at end of any such experience (by Mentors) Course/Project Skills Assessment Form at end of every course (by all Professors/Instructors) Student Evaluation of Teaching (SET) At end of every course (by all students) Video of Presentations at Design/Project Presentations (by all Professors/Instructors)

Program Assessment Ethics Integration Assessment Form at end of CAPSTONE Courses (by CAPSTONE Professors) Graduating Student Exit Survey (Part I) at end of CAPSTONE Courses (by CAPSTONE Professors) Graduating Student Exit Survey (Part II) at Graduation time (by Department) Internship Assessment Form (Student) at completion of all Internships (by Mentors) Internship Assessment Form (Mentor) at completion of all Internships (by Mentors) COOP Supervisory Evaluation Form at completion of COOP terms (by Mentors) COOP Student Evaluation Form at completion of COOP terms (by Mentors) Student Resume (Special Format)** start at UNIV-0004 Course; maintain up-to-date thru college years

Post Graduation Alumni Survey mail to alumni, every 3 years (by Department) Employers Survey mail to employers, every 3 years (by Department)

OUTCOMES ASSESSMENT MATRIX Criterion 3 (a-k Outcomes) or Program Outcomes

Assessment Tools

a

b

c

d

e

f

g

h

i

j

k

Pre-Engineering Freshman Orientation Questionnaire X X X X X X Ethics Integration Assessment Form X

Course Assessment Laboratory Reports (copies of) X X X X X X Exams, Quizzes, Homeworks (copies of) X X X X X X X EIT Exam Statistical Report X X X Written Report Evaluation X X X Oral Presentation Assessment X X Teamwork Assessment Form (I) X Teamwork Assessment Form (II) X Peer Evaluation Form X Undergrad Research Exp Assessm. Form X X X X Course/Project Skills Assessment Form X X X X X X X X X X X Student Evaluation of Teaching (SET) X X X X X X X X X X Video of Presentations X X X X X X

Program Assessment Ethics Integration Assessment Form X Graduating Student Exit Survey (Part I) X X X X X X X X X X X Graduating Student Exit Survey (Part II) X X X X X X X X X X X Internship Assessment Form (Student) X X X Internship Assessment Form (Mentor) X X X COOP Supervisory Evaluation Form X X X COOP Student Evaluation Form X X X Student Resume (Special Format)** X X X X X X X X X X X

Post Graduation Alumni Survey X X X X X X X X X X X Employers Survey X X X X X X X X X X X

You don’t have to evaluate everything all the time!

Assessment planning @ NEU

Assessment tool To be completed by Timeline Responsible Pre-Survey (all learning outcomes + reflection)

Students First day of class Provost office

Exit-Survey (all learning outcomes + faculty evaluation + reflection)

Students, Faculty, Alumni, Employers

Last day of class, every three years (for alumni & employers)

Provost office

Hmwk, tests, projects Students All courses Faculty Teamwork assessment Students, Peers Once per year - tbd Faculty Oral presentation assessment

Students, Peers All courses Faculty

Written report assessment

Students All courses Faculty

Networking Focus Group

Selected group of students

Once per year - tbd Faculty

Ethics assessment Students, peers Once per year - tbd Faculty Project management assessment

Students, peers Once per year - tbd Faculty

Diversity Focus Group Selected group of students; selected group of faculty

Once per year – tbd Provost Office

Adaptability Focus Group

Selected group of students; Selected group of faculty

Once per year – tbd Provost Office

Assessment planning in the Informatics BEng

ASIGNATURA

Prueba escrita de respuesta

abierta

Prueba objetiva

(tipo test)

Trabajo académico Proyecto Examen Oral Autoevaluación Coevaluación Mapa

Conceptual

Oneminute paper

caso Portafolio Observación Diario

PRI

MERO

Análisis matemático 60% 20% 20%Fundamentos de computadores 50% 10% 20% 20%Fundamentos Físicos de la Informática 50% 20% 20% 20%Introducción a la informática y a la programación 80% 20%Matemática discreta 90%Algebra 90% 10%Estadística 55% 35% 10%Fundamentos de organización de empresas 40% 36% 24%Programación 80% 20%Tecnología de computadores 20% 42,5% 20% 12,5% 5%

SEGUNDO

Fundamentos de sistemas operativos 70% 10% 20%Lenguajes, tecnologías y paradigmas de la programación 30% 60% 10%

Teoría de autómatas y lenguajes formales 70% 10% 20%Estructura de computadores 85% 15%Redes de computadores 60% 40%Concurrencia y sistemas distribuidos 60% 20% 20%Deontología y profesionalismo 25% 10% 18% 3% 5% 40%Estructuras de datos y algoritmos 80%Interfaces persona computador 20% 40% 20% 20%

TERCER

O

Arquitectura e ingeniería de computadores 88% 12%Bases de datos y sistemas de información 80% 20%Computación paralela 60% 15% 15% 10%Ingeniería del software 40% 35% 5% 5% 15%Sistemas inteligentes 35% 46% 20%Tecnología de sistemas de información en la red 75% 25%Gestión de proyectos 20% 20% 40% 20%

com

pulso

ry su

bjec

ts

Eduardo Vendrell, Dean of Informatics @ UPV

courses

Assessment tools

Think-Pare-Share Exercise

• Make a list of assessment tools you currently use to evaluate student learning outcomes/competencies.

• Make a list of assessment tools you need to evaluate student learning outcomes/competencies.

5 Steps• Define and validate your program outcomes/competencies• Plan (MAP) the learning process• Measure & analyze learning outcomes• Share & discuss results• Make decisions and re-engineer to improve

next time!

Copyright © 2014 IonIdea Inc.

ionize curriculum design for outcome based education system

v3.0

www.ioncudos.com

Copyright © 2014 IonIdea Inc.

OUTCOMES BASED CURRICULUM MANAGEMENT SYSTEM

Curriculum Design

Curriculum Delivery Assessment

Delivery Planning

Assessment Planning

Import of data for Attainment

Copyright © 2014 IonIdea Inc.

MODULES

challenges & opportunities in implementing outcomes based education

Challenges• Inertia • Fear• Resources• Time

Opportunities• Awareness of the need (why)• Capacity building (how)• Community (who)• Start small, test, grow• Document & share outcomes

change management (next time)!

Can we change?

Albert Einstein

“The value of an education is not the learning of many facts, but the training of the mind to think something that cannot be learned from textbooks.”

¡Gracias!Thank you!lueny.morell@gmail.comwww.luenymorell.com (blog)www.innnovahied.com

In 2016Inaugurating the IGIP

International Engineering EducatorTraining Center