Developing a Measurement Instrument to Assess …...2019/09/06 · THE CURRENT SITUATION Ngai, S....

DEVELOPING A MEASUREMENT INSTRUMENT TO ASSESS STUDENT LEARNING OUTCOMES

AFTER SERVICE-LEARNING EXPERIENCE Calvin LAU

1

THE PROJECT BACKGROUND

A Cross-institutional Project Funded by the UGC

To Enhance and support the development of S-L as an effective pedagogical strategy at the participating UGC-funded institutions and beyond through a cross-institutional multi-faceted professional staff development platform

e-Learning module

Measurement instruments

Community of Practice

Action research projects

Good practice database

2

THE MEASUREMENT INSTRUMENT GOALS

Develop a valid and parsimonious measurement instrument

to assess comprehensively student learning outcomes

after service-learning experience

3

Student

Learning

Outcomes

Community ImpactService-Learning

Management

THE CURRENT SITUATION

A standardized and validated

Measurement instrument for

Student learning outcomes in

Service-leanring?

4


Toncar, M. F., Reid, J. S., Burns, D. J., Anderson, C. E., & Nguyen, H. P. (2006). Uniform assessment of the benefits of service learning: the development, evaluation, and implementation of the SELEB scale. The Journal of Marketing Theory and Practice, 14(3), 223–238. https://doi.org/10.2753/mtp1069-6679140304.

5

https://doi.org/10.2753/mtp1069-6679140304


Ngai, S. S. (2009). The effects of program characteristics and psychological engagement on service-learningoutcomes: A study of university students in Hong Kong. Adolescence, 44, 375–389. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/19764273.

Domain: Personal development, civic involvement, agency support, involvement with service users, class experience, psychological engagement

Scales, P. C, Blyth, D. A., Berkas, T. H., & Kielsmeier, J. C. (2000). The effects of service-learning on middle school students' social responsibility and academic success. Journal of Early Adolescence, 29(3), 332-358.

Bringle, R. G., Phillips, M. A., & Hudson, M. (2004). The measure of service learning: Research scales to assess student experiences. Washington, DC: American Psychological Association.

6

https://www.ncbi.nlm.nih.gov/pubmed/19764273


7

Questionnaire BU EdUHK LU PolyU

Student Learning Outcomes Measurement Questionnaire - - Y -

Common Outcomes Measurement Y - Y -

Global Citizenship Scale - - Y Y

Intercultural Effectiveness Scale - - - Y

Student Experience Questionnaire Y - - Y

Leadership Qualities Assessment Rubric Y - - -

The Scale Measuring Educators' Beliefs about Diversity in Personal and Professional Contexts - Y - -

Co-curricular and Service Learning - Y - -


Ma, H. K. C., Chan, C. W. F., Tse, P. H., & Chan, C. M. A. (2016). A common outcomes measurement for service-learning in Hong Kong. Unpublished manuscript.

Measure by 9 domains:

1. Self-understanding/ Confidence

2. Communication Skills

3. Problem-solving Skills

4. Civic Engagement, Social Responsibility and Willingness to Contribute

5. Team Skills

6. Self-Reflection

7. General Knowledge Application

8. Caring for Others

9. Intercultural Competence

8

WHAT WE DID

Literature review on student learning outcomes and the Conceptual Framework

Understanding knowledge

Applying knowledge

Relationships skills

Team work

Problem solving skills

Critical thinking skills

Creativity

Sense of social responsibility

Commitment to social service

Understanding community

Empathy

9

WHAT WE DID

The proposed theoretical model for the measurement instrument

10

Conceptual

Category

Knowledge

Application

Personal and

Professional Skills

Civic Orientation and

Engagement Self-awareness

Learning

Outcome Domain

1. Knowledge

Application

2. Relationship Skills 8. Sense of Social Responsibility 13. Self-

understanding

3. Team Skills 9. Commitment to Social Betterment 14. Commitment to

Self-improvement

4. Problem-solving Skills 10. Understanding Community 15. Self-esteem

5. Critical-thinking Skills 11. Respecting Diversity

6. Self-reflection Skills 12. Empathy and Caring for Others

7. Creativity

WHAT WE DID

Methods for the measurement instrument development

11

Item generation Item review Pilot test Validation test

Relevant scales

Scales being used

Brainstorm

By the panel

By the subject

matter experts

Item readability

Administration

hiccups

Panel reviewed

feedback and

revised items

Psychometric

properties

Underlying

dimensionality

Reliability

Item reduction

Translation

WHAT WE DID

Pilot test

83 university students invited male: 34.9%, female: 65.1%

mean age: 20.5

1 hour (40 mins: answering; 20 mins: discussing)

35 items revised or simplified in wording

2 items discarded

12

Examples of item revision:

Original: I always actively discuss betterments for our

community.

Revised: I always actively discuss possible improvements

for our community.

Original: I consider contextual factors when reflecting on

how well I have performed

Revised: I consider circumstances when reflecting on how

well I have performed.

WHAT WE DID

Validation test – EFA

Analysis: the MAP Test: to uncover the optimal number of factors

EFA: to uncover the underlying dimensionality for each category

item exclusion criteria in EFA:

items with the absolute value of the factor loading lower than .40

double-loaded items

item-total correlation

semantic consideration

Cronbach’s Alpha: to obtain reliability index

13

WHAT WE DID

Validation test – EFA


mean age: 20.9

Arts: 23.4%; Social Science: 15.6%; Business: 22.4%; Engineering & Science: 27.5%; Nursing: 11.1%

previous service-learning experience: 70.5%

1 hour

14

WHAT WE FOUND

The MAP test

15

Category Optimal Number of Factors

Knowledge Application 1

Personal and Professional Skills 5

Civic Orientation and Engagement 4

Self-awareness 3

WHAT WE FOUND

EFA and reliability results (knowledge application: 1-factor solution)

16

No Item

Absolute Value of

Factor Loading

Item-Total

Correlation

1 I know how to apply what I learn in class to solve real-life problems. .881 .771

2 I am able to apply/integrate classroom knowledge to deal with complex

issues.

.867 .752

3 I know how to transfer knowledge and skills from one setting to another. .853 .731

4 I can make connections between theory and practice. .799 .656

Variance explained 72.35% (α = .872)

WHAT WE FOUND

EFA and reliability results (personal and professional skills: 4-factor solution)

17

Absolute Value of Factor Loading

No Item

Creative

Problem

Solving Skills

(α =.919)

Relationship

and Team

Skills

Self-

reflection

Skills

Critical

Thinking

Skills

Item-Total

Correlation

1 I am not afraid of trying new things. .867 .610

2 I am able to generate original ideas. .685 .700

3 I am able to solve challenging real-life problems. .652 .783

4 I feel confident in dealing with a problem. .635 .747

5 When necessary, I can think of alternatives. .534 .764

6 I feel confident in identifying the core of a problem. .518 .771

7 I am able to look at an issue from a fresh perspective. .511 .720

8 I often modify my strategies to solve a problem when the

situation changes.

.472 .744

WHAT WE FOUND


18


No Item

Creative

Problem

Solving Skills

Relationship

and Team

Skills

(α =.925)

Self-

reflection

Skills

Critical

Thinking

Skills

Item-Total

Correlation

9 I am good at keeping in touch with people. .886 .691

10 I am good at building relationships between people. .730 .691

11 I can build long-term relationships with people. .716 .711

12 I can easily establish effective relationships with people. .706 .749

13 I am good at resolving conflicts. .649 .733

14 I am confident in leading others toward common goals. .543 .731

15 I participate effectively in group discussions and

activities.

.531 .761

16 I have the necessary skills for making groups or

organizations function effectively.

.470 .764

WHAT WE FOUND


19


No Item

Creative

Problem

Solving Skills

Relationship

and Team

Skills

Self-

reflection

Skills

(α =.848)

Critical

Thinking

Skills

(α =.751)

Item-Total

Correlation

17 I will evaluate myself after completing a task. .838 .678

18 I reflect on myself regularly. .766 .653

19 I always think how I can improve myself. .633 .651

20 I consider circumstances when reflecting on how well I

have performed.

.542 .710

21 I can analyze an issue comprehensively. .732 .601

22 I often look at complex issues from different angles. .654 .655

23 I can understand others' viewpoints when we are

making decisions together.

.411 .627


WHAT WE FOUND

EFA and reliability results (civic orientation and engagement: 3-factor solution)

20


No Item

Community Commitment and

Understanding(α =.919)

Caring and

Respect

Sense of Social

Responsibility

Item-Total Correlation

1 I always actively discuss possible improvements for our community.

.861 .584

2 I can identify useful resources of a community. .822 .7243 I think about how I can serve the community after graduating. .733 .6394 I can identify challenges in the community. .727 .7095 I can investigate the challenges faced by people in need in a

community..726 .726

6 I will contribute my abilities to make the community a better place.

.692 .735

7 I can identify issues that are important for a disadvantaged community.

.675 .726

8 I will play my part to reduce social problems. .608 .719

WHAT WE FOUND


21


No Item

Community

Commitment and

Understanding

Caring

and

Respect

(α =.907)

Sense of

Social

Responsibility

Item-Total

Correlation

9 I respect the needs of people from different backgrounds. .795 .645

10 I appreciate the ideas of people from different backgrounds. .789 .693

11 I am willing to try to understand people whose background is

different from mine.

.751 .736

12 I can respect people whose background is different from mine. .705 .576

13 I consider others' points of view. .685 .690

14 I care about others. .478 .746

15 I observe others' feelings and emotions. .467 .692

WHAT WE FOUND


22


No Item

Community

Commitment and

Understanding

Caring

and

Respect

Sense of

Social

Responsibility

(α =.813)

Item-Total

Correlation

16 I believe that everybody should be encouraged to participate in

civic affairs.

.789 .622

17 I believe that taking care of people who are in need is

everyone’s responsibility.

.750 .681

18 I feel obligated to help those who are less fortunate than me. .605 .700


WHAT WE FOUND

EFA and reliability results (self-awareness: 3-factor solution)

23


No Item

Self-

efficacy

(α =.859)

Self-

Understanding

(α =.845)

Commitment to

Self-improvement


1 I am satisfied with my achievement so far. .842 .644

2 Most things I do, I do well. .830 .694

3 I have many good qualities. .770 .761

4 I am positive about myself. .736 .721

5 I know my strengths and weaknesses. .901 .639

6 I have a clear picture of what I am like as a person. .877 .642

7 I have a clear understanding of my own values and

principles.

.631 .743

8 I know what I need in my life. .527 .700

WHAT WE FOUND

EFA and reliability results (self-awareness: 3-factor solution)

24


No Item

Self-

efficacy

Self-

Understanding

Commitment to

Self-improvement

(α =.829)


9 I look out for new skills or knowledge to acquire. .941 .641

10 I am always motivated to learn. .762 .667

11 I always keep my knowledge and skills up-to-date. .660 .736


WHAT WE FOUND

Summary – EFA

good results with clear factor structure knowledge application

personal and professional skills

civic orientation and engagement

self-awareness

higher order domains identified creative problem-solving skills

relationship and team skills

caring and respect

community commitment and understanding 25

WHAT WE FOUND

Summary – EFA

Presented the findings and the measurement instrument at the 7th Asia-Pacific Regional Conference Service-LearningSnell, R. S. & Lau, K. H. (2019). Developing a Measurement Instrument to Assess Student Learning

Outcomes after Service-Learning Experience. 7th Asia-Pacific Regional Conference on Service-Learning.

https://www.suss.edu.sg/aprcsl-registration/papers/119-PPR127.pdf

26

https://www.suss.edu.sg/aprcsl-registration/papers/119-PPR127.pdf

WHAT WE DID

Validation test – CFA

Analysis: CFA: to confirm the underlying dimensionality for each category obtained from EFA

Using EQS 6.4

Method: ML


27

WHAT WE DID

Validation test – CFA


mean age: 20.5

backgrounds: Arts: 12.7%; Social Science: 14.3%; Business: 19.9%; Engineering & Science: 40.9%;

Healthcare: 12.2%


Separate CFA for each category

28

WHAT WE FOUND

29

v

v

Knowledge application

WHAT WE FOUND

30

Personal and professional skills

WHAT WE FOUND

31

Civic orientation and engagement

WHAT WE FOUND

32

Self-awareness

WHAT WE FOUND

Summary – CFA

WHAT WE DID

Validation test – Test-retest

Analysis: Test-retest: to confirm the domain’s scores are stable over time without manipulation

SPSS, ICC index


34

WHAT WE DID

Validation test – Test-retest


mean age: 20.0

backgrounds: Arts: 5.8%; Social Science: 5.8%; Business: 11.5%; Engineering & Science: 65.4%;

Healthcare: 11.5%


35

WHAT WE FOUND

Summary – Test-retest

WHAT NEXT

Further validation

Pretest-posttest validation

Sep to Dec 2019 Participating Institution:

EdUHK, HKBU, LU, PolyU

The Hang Seng University of Hong Kong

Republic Polytechnic of Singapore

Followed by translation into Chinese in early 2020

Target: to be available from 2020 onwards37

WHAT NEXT

What would be your input:

Arrange your classes to answer the questionnaires: before your S-L project and training start

after the S-L project finishes

Answer an instructor’s questionnaire to prioritize the expected learning outcome

That’s it. Simple!

38

39

Thank You!

Calvin LAU

: [email protected]

Robin Stanley SNELL

: [email protected]

Questions Comments

mailto:[email protected]

mailto:[email protected]

Developing a Measurement Instrument to Assess …...2019/09/06 · THE CURRENT SITUATION Ngai, S....

Documents

Transcript of Developing a Measurement Instrument to Assess …...2019/09/06 · THE CURRENT SITUATION Ngai, S....