Assessment Valle

7/29/2019 Assessment Valle

1/5

Assessment of student performance in problem-basedlearning tutorial sessions

Rosamar{a Valle, lleana Petra, Adrian Mart{nez-Gonzalez, Jose Antonio Rojas-Ramirez,

Sara Morales-Lopez & Beatriz Pin~

a-Garza

1,21,2 Objectives To assess student performance during tuto-

rial sessions in problem-based learning (PBL).

Design A 24-item rating scale was developed to assess

student performance during tutorial sessions in prob-

lem-based learning (PBL) as conducted during the

pre-clinical years of Medical School at the National

Autonomous University of Mexico. Items were divided

into three categories: Independent study, Group inter-action and Reasoning skills. Fourteen tutors assessed

152 rst and second-year students in 16 tutorial

groups. An exploratory factor analysis with an Oblimin

rotation was carried out to identify the underlying

dimensions of the questionnaire.

Setting Medical School at the National Autonomous

University of Mexico.

Subjects Medical students.

Results Factor analysis yielded four factors (Indepen-

dent study, Group interaction, Reasoning skills, and

Active participation) which together accounted for

766% of the variance. Their Cronbach reliability

coefcients were 095, 083, 094 and 093, respectively,

and 096 for the scale as a whole.

Conclusions It was concluded that the questionnaire

provides a reliable identication of the fundamentalcomponents of the PBL method as observable in

tutorial groups and could be a useful assessment

instrument for tutors wishing to monitor students'

progress in each of these components.

Keywords Problem-based learning; medical students;

undergraduate medical education, *standards; educa-

tional measurement, *standards.

Medical Education 1999;33:818822

Introduction

The Faculty of Medicine at the National Autonomous

University of Mexico (UNAM), founded in 1579, is the

oldest medical school in America. It has a total

undergraduate population of 5570 students and annu-

ally admits approximately 950 students. The under-

graduate programme has a duration of six years (the

rst two include mainly basic sciences followed by two

years of clinical medicine, a year of internship and a

year of medical practice in a community. A traditional,

lecture-based approach is generally used in most of the

courses. However, the latest curriculum revision, car-

ried out in 1993, suggests the use of innovative teaching

methods in medicine, such as problem solving, tutorial

teaching, and interdisciplinary learning.1

Educational strategies based on problem-basedlearning methods have been used in some of the courses

in the rst four years with the following objectives: (a)

introducing a self-learning method using a problem-

solving approach (b) promoting active student

participation and (c) helping students to acquire rea-

soning and group interaction skills. To achieve these

goals, students assemble in tutorial groups of nine

assisted by one staff tutor and are given a medical prob-

lem to analyse. The rst session begins with a case pre-

sentation followed by clarication of concepts and facts

that are new to the students. Students are then asked to

identify and analyse the problem, and attempt to for-

mulate hypotheses based on their previous knowledge.

After this, students determine learning objectives and

resources required to obtain the necessary information.

After a period of 3872 h during which they locate and

collect information, students meet with their tutor for a

second session to present their ndings, analyse the in-

formation they have gathered and discuss the case.

This method is based on problem-based learning

strategies as proposed by Schmidt,2 involving the

Facultad de Medicina, Direccio n General de Evaluacion Educativa,

Coordinacio n del Sistema de Universidad Abierta y Educacion a

Distancia, Universidad Nacional Auto noma de Mexico.

Correspondence: Dr Rosamara Valle. Direccio n General de Evaluacion

Educativa. Edicio de los Consejos Academicos, Circuito Exterior,

Ciudad Universitaria, Mexico, DF, 04510

Research papers

818 Blackwell Science Ltd MED ICA L ED UCA TI ON 1999;33:818822


2/5

following seven steps: (l) clarifying terms and concepts

not readily comprehensible; (2) dening the problem;

(3) analysing the problem; (4) drawing up a systematic

inventory of the explanations deduced from step three;

(5) formulating learning objectives; (6) collecting

additional information outside the group and (7) syn-

thesizing and testing the newly acquired information.The UNAM's Faculty of Medicine has feltit necessary

to assess students' progress during tutorial sessions and

the achievement of PBL objectives. According to Hay,3

the evaluation of PBL students should be based on actual

performance in their learning setting. This is where they

should learn to make relevant hypotheses, identify their

learning needs, use appropriate learning resources,

provide evidence of learning and act as a responsible

team member of the tutorial group. However, it has

proved difcult for tutors to judge whether their students

have acquired the desired competencies based on evi-

dence obtained during tutorial sessions. Some tutors

tend to ask open-ended questions which, since they are

not very specic, provide little opportunity for accurately

determining students' individual levels of understanding

or to make comparisons.4 Some instruments evaluate

group learning skills, knowledge and critical thinking;5

others refer to responsibility, information acquisition,

communication, critical sense and self-assessment.

The extensive reviews of PBL carried out by

Albanese & Mitchell6 and Vernon & Blake7 do not

include studies related to the assessment of student

performance in tutorial sessions. Des Marchais & Vu8

assessed tutor rating instruments for evaluating stu-

dents' communication, reasoning and group interactionskills, autonomy and humanism. They concluded that

the validity of these types of instruments is still psy-

chometrically questionable.

With the exception of Hebert & Bravo's Tutotest

(1996) no other standardized instruments were found

that are designed to assess medical students' attitudes

and skills acquired as a result of participating in PBL

tutorials. Through factor analysis of their 44-item

questionnaire, the authors identied four factors:

effectiveness within the group, communication and

leadership skills, scientic curiosity and respect for

peers. Despite the relatively low response (675%) and

tutor participation (70%) rates, and variation in the

number of times that some of the students were eval-

uated, the test was found to be a reliable and practical

instrument, useful for evaluating students' skills and

attitudes during tutorials.

The aim of this study was to develop an instrument

to assess student performance as a result of participa-

tion in PBL tutorial sessions, and evaluate its construct

validity and reliability. It is part of a project that

includes the development of instruments and methods

to evaluate PBL outcomes at UNAM's Faculty of

Medicine.

Methods

A team of medical teachers, psychologists and educa-tion specialists was established to develop an instru-

ment enabling tutors to assess students' attitudes and

competencies acquired through their participation in

PBL tutorial sessions. This team included physicians

trained in PBL educational strategies at the Schools of

Medicine at McMaster University, Michigan State

University and the University of Limburg, Maastricht.

These physicians trained and supervised the rest of the

team as well as other teachers applying PBL.

The behaviours to be analysed were closely linked to

those involved in PBL2,911 and the UNAM faculty's

experience in PBL. Such behaviours were divided into

three categories: Independent study, Group interaction

skills and Reasoning skills.

These categories were used as frame of reference for

selecting the items to be included in the questionnaire.

Certain criteria had to be met: all items should describe

behaviours related to PBL, be observable by the tutor

during the sessions, be stated in positive terms and

unambiguously, and clearly belong to only one cate-

gory. In keeping with this procedure, 39 items were

written and their face validity analysed by a group of 12

teachers who were asked to assign each item to a cat-

egory according to its content. Of these, 21 were

accepted by at least 11 teachers, eight had to berewritten and 10 were eliminated because they were

apparently not relevant or their content had already

been included in other items. The degree of clarity of

the 29 remaining items was rated on a ve-point scale

by ve independent referees familiar with the PBL

method. As a result, minor corrections were made in six

of the items. Lastly, these 29 items were included in the

questionnaire in random order. They describe behav-

iours that the tutor has to rate on a six-point scale

ranging from never (1) to always (6) to indicate current

frequency.

The tutors who participated in this study were

teachers at the Faculty of Medicine who had partici-

pated in training sessions on PBL fundamentals, strat-

egies and evaluation. In addition, they had been using

this method of instruction for the previous two years and

were trained in the use of this instrument. Assessments

were performed at the end of the second tutorial session.

An exploratory factor analysis was carried out to

identify the underlying aspects. A principal component

analysis was followed by an Oblimin rotation. This

Blackwell Science Ltd MED ICA L ED UCA TI ON 1999;33:818822

Assessment of student performance in PBL tutorial sessions R Valle et al. 819


3/5

procedure was chosen on the assumption that such

components would not be independent.12,13

Results

Fourteen tutors evaluated 152 rst- and second-year

students in 16 tutorial groups. The sample was madeup of 483% men and 517% women with a mean age of

19 years. All students attended PBL tutorials for at

least 12 weeks before the study and agreed to partici-

pate in the study.

The questionnaire was given once after each second

tutorial group session and all 14 tutors returned the

questionnaires. However, given that only fully com-

pleted questionnaires could be included in the analysis,

12 had to be eliminated, meaning that 140 question-

naires remained for analysis (Table 1).

The initial factor analysis of the 29 item scale showed

that two items had low load values (< 040) while three

loaded substantially on more than one factor (< 045).

These ve items were therefore eliminated. Factor

analysis of the remaining 24-item pool yielded 4 factors

with eigenvalues3 < 10 which together accounted for

766% of the variance, with each one accounting for

539%, 114%, 59% and 54%, respectively (Table l).

On the basis of their content, factors were designated asIndependent study, Group interaction, Reasoning skills

and Active participation. Their Cronbach reliability

coefcients were 095, 083, 094 and 093, respectively,

and the total scale reliability was 096.

The rst factor, Independent study, comprises nine

items linked to students' initiative, motivation and

persistence in searching for information, studying and

achieving the learning objectives and tasks agreed on by

the group. Inter-item correlations of this subscale

ranged from 057 to 091. The ve items referring to

factor 2 (Group interaction) concern students' abilities

to function in a group, such as openness to suggestions

Table 17 Factor analysis of students performance in PBL tutorial sessions questionnaire*

Factor loads

Items

Factor 1

Independent

study

Factor 2

Group

interaction

Factor 3

Reasoning

skills

Factor 4

Active

participation

25. Uses different resources to obtain needed information 0968 A 0004 0103 0012

26. Uses additional references to those suggested by the programme 0836 A 0048 0011 A 0096

21. Presents well-organized information relevant to the case 0824 0128 A 0145 0298

28. Is persistent in the study of the case 0812 A 0120 A 0138 A 0132

27. Is motivated to know more 0808

A0

124

A0

107

A0

134

24. Implements activities to achieve the learning objectives 0775 0062 A 0074 A 0161

29. Shows initiative in the study of the case 0717 A 0160 A 0215 A 0158

23. Shows initiative in the search of information 0706 0067 A 0113 A 0193

9. Accomplishes tasks agreed by the group 0653 0195 0092 0012

5. Accepts suggestions about his/her work A 0132 0813 0086 A 0306

2. Adjusts to different group roles 0056 0777 A 0175 A 0023

1. Shows respect to his/her peers 0101 0761 A 0130 0217

11. Accepts decisions made by the group A 0066 0755 A 0119 A 0015

6. Listens attentively to other members of the group 0146 0689 0157 A 0094

19. Identies his/her learning needs A 0055 0146 A 0882 0187

16. Asks questions related to the case 0025 A 0017 A 0812 A 0198

20. Accepts or rejects hypothesis on well-based foundations 0148 0043 A 0769 A 0055

15. Analyses different components of the case 0180 A 0087 A 0669 A 0285

13. Claries facts, concepts and terminology 0209 0

022

A0

578

A0

307

14. Identies information relevant to the case 0299 A 0021 A 0498 A 0316

7. Gives feedback (reections, ideas and suggestions) to the group 0105 0253 A 0073 A 0741

3. Helps her/his peers to clarify ideas 0200 0141 A 0027 A 0740

4. Participates in case discussions 0197 A 0029 A 0275 A 0639

8. Shares knowledge with the group 0171 0136 A 0244 A 0627

Eigenvalues 1293 274 142 129

Contribution to variance (%) 539 114 59 54

* Data derive from observations on 140 students made by 14 staff tutors.

Assessment of student performance in PBL tutorial sessions R Valle et al.820



4/5

and decisions, adjustment to different group roles and

respect towards their peers. Inter-item correlations

range from 057 to 066. The six items included in

factor 3 (Reasoning skills) are related to students' abi-

lity to analyse cases, formulate hypotheses, and clarify

concepts. The inter-item correlations ranged from 065

to 088. The last factor was called Active participation

because the four items included reect a specic form

of interacting with the group that includes behaviours

such as contributing, helping and sharing reections,

ideas and knowledge. Here inter-item correlations

ranged from 069 to 083.

Pearson product-moment correlations were com-

puted to assess the interrelations between the four

factors (Table 2). Although all correlations were signi-

cant (p < 0000, one-tail test), the highest were found

between Reasoning skills and Independent study, and

between Active participation and Reasoning skills andIndependent study.

Discussion

The implementation of educational innovations in the

curriculum, such as problem-based learning, makes it

essential to evaluate processes and outcomes and to

develop the necessary instruments to achieve this

goal. As part of these evaluation objectives, it was

decided to elaborate and assess a standardized

questionnaire that could be used by PBL tutors to

monitor students' progress in the achievement of PBL

objectives.

Results of the analysis of the questionnaire yielded

four factors reecting essential components of problem-

based learning: Independent study, Group interaction,

Reasoning skills, and Active participation.2,911

Identication of these aspects enables tutors to assess

PBL components separately or as a whole, as required.

They may nd it useful for assessing their students

during the process (formative evaluation) or at the end

of the course or period (summative evaluation), or for

evaluating the extent to which changes in attitudes and

target skills are being achieved through this educational

strategy.

The rst factor (Independent study) reveals an

underlying strategy crucial to PBL; the training of

students to manage their own learning. Two of theitems (9 and 21) included in this factor had originally

been assigned to the Group interaction and Reasoning

skills categories, respectively. However, the content of

these two items would seem to be more the result of

Independent study than of the other factors. The sec-

ond factor (Group interaction) permits the identica-

tion of fundamental attitudes for team work, another

essential component of PBL. The ability to analyse

problems and propose or reject well-founded hypothe-

ses is another fundamental aim of PBL expressed in the

items comprising Reasoning skills.

Finally, it is worth pointing out the distinction made

between two factors: Group interaction and Active

participation in the group suggest attitudes of co-op-

eration and collaboration in the process of analysing the

problem. The items included in the Group interaction

and Active participation factors were originally con-

ceived as part of a single category, but results have

shown that two different aspects are involved. At the

same time, the fact that the tutor is in a position to

choose the type of group interaction he wishes to en-

courage will allow him to foster collaboration instead of

competition, which is in keeping with the policies of

both McMaster University14,15 and UNAM'S Medi-

cine Faculty, which seek to encourage team workamong students. On the other hand, it has been noted

that the tutor should not intervene in his students

evaluation since this may alter the social structure of a

tutorial group.16 However, to determine the nature and

extent of this inuence it would be necessary to carry

out more research on the subject.

The results of the correlation analysis show that the

factors are signicantly interrelated, particularly Inde-

pendent study, Reasoning skills and Active participa-

tion. Conversely, there is only slight correlation

between Group interaction and Independent study,

Reasoning skills and Active participation. These two

sets are linked to intellectual activities and socio-affec-

tive activities, respectively, both crucial to the achieve-

ment of PBL objectives.

The results obtained conrm the construct validity of

the questionnaire and the reliability (internal consis-

tency) of each factor. Unlike the questionnaire devised

by Hebert & Bravo,17 the items comprising each factor

only have a high factor load in one of them. However,

this study constitutes the initial phase of the

Table 2 Factors correlation matrix

Factors

Items 1 2 3 4

Factor 1: Independent studyFactor 2: Group interaction 0343*

Factor 3: Reasoning abilities 0767* 0348*

Factor 4 Active participation 0734* 0430* 0739*

n = 140.

*p < 0000, one-tail test.

Assessment of student performance in PBL tutorial sessions R Valle et al. 821



5/5

psychometric evaluation of the questionnaire. Further

studies are indicated not only to corroborate the results

obtained with the questionnaire but also to determine

the link between PBL and students' performance in the

acquisition of knowledge and other skills required for

physicians' training.

Lastly, the questionnaire may be used for the fol-lowing purposes: to track the evolution of attitudes and

skills during tutorials, to serve as a tool for providing

useful feedback for students and to evaluate overall

tutorial group performance at the end of the course.

Acknowledgements

We would like to thank Dr Enrique Pin a for his support

to this project.

References

1 De la Fuente JR, Pin a-Garza E, Gutierrez-Avila JH. La

formacio n del medico del siglo XXI y el Plan Unico de

Estudios de la Facultad de Medicina de la Universidad Nac-

ional Auto noma de Mexico. Educacion Medica Y Salud

1994;28:33140.

2 Schmidt HG. Problem-based learning: rationale and descrip-

tion. Med Educ 1983;17:116.

3 Hay J. Tutorial performance. In: Evaluation Methods:

a Resource Handbook (eds G Norman, S Shannon.). Hamilton

Ontario: McMaster University, 1995.

4 Graesser A, Person N. Questions asked during tutoring. Am

Educational Res J 1994;31:10437.

5 Woodward CA. Some reections on evaluation of outcomes of

innovative medical education programmes during the practice

period. Ann Community-Oriented Educ 1992;5:18191.

4

6 Albanese M, Mitchell S. Problem-based learning: a review of

literature on its outcomes and implementation issues. Acad

Med 1993;68:5281.

7 Vernon D, Blake R. Does problem-based learning work?

A meta-analysis of evaluative research. Acad Med

1993;68:55063.

8 Des Marchais JE, Vu NV. Developing and evaluating the

student assessment system in the pre-clinical problem-based

curriculum at Sherbrooke. Acad Med 1996;71:27483.

9 Barrows HS, Tamblyn RM. Problem-Based Learning: an

Approach to Medical Education. New York: Springer, 1980.

10 Barrows HS. How to Design a Problem-Based Curriculum for the

Pre-Clinical Years. New York: Springer, 1985.

11 Walton HJ, Matthews MB. Essentials of problem based

learning. Med Educ 1989;23:54258.

12 Tabachnik B, Fidell S. Using Multivariate Statistics. New York:

Harper & Row, 1983.

13 Floyd F, Widaman K. Factor analysis in the development and

renement of clinical assessment instruments. Psychol Assess-

ment 1995;7:28699.

14 Pallie W, Carr DH. The McMaster medical education phi-

losophy in theory, practice and historical perspective. Med

Teacher 1987;9:5971.

15 Neufeld VR, Barrows HS. The McMaster philosophy: an ap-

proach to medical education. J Med Educ 1974;49:104050.

16 Van der Vleuter C, Verwijnen M. A system for student as-

sessment. In: C Van der Vleuter, W Wijnen. eds. Problem

Based Learning: Perspectives from the Maastricht Experience.

Amsterdam: CIP DATA Koninklijke Bibliotheek, Den Haag,

pp 2749, 1990.5

17 Hebert R, Bravo G. Development and validation of an eval-

uation instrument for medical students in tutorials. Acad Med

1996;71:48894.

Received 21 July 1997; editorial comments to authors 28 October

1997; accepted for publication 22 January 19996

Assessment of student performance in PBL tutorial sessions R Valle et al.822


Assessment Valle

Documents

Transcript of Assessment Valle