1

Student-Oriented Meta-Assessment in a Project-Based

Systems Engineering Course

Niva Wengrowicz1

Yehudit Judy Dori1,2

Dov Dori1,2

1-Technion, Israel Institute of Technology2-MIT, Massachusetts Institute of Technology

3

The Goal of Engineering Education

engineering education should promote among other things,

teamwork and project-based learning (PBL)

Modern engineering education programs aim to enrich students with the necessary

knowledge, skills, and attitudes for becoming successful young engineers

NSF determined that

4

Project-Based Learning (PBL)• The PBL approach emphasizes students' project

construction to represent what they learn.

• PBL is characterized by:– Active learning– No single correct solution, but rather

an open-ended driving challenge– Collaboration among peers• Our course is based on students’ projects of constructing

conceptual modeling

6

Conceptual Models

• In systems engineering, a conceptual model is necessary for:

– coherent thinking– sharing ideas– providing common ground for communication– solving problems jointly.

In our course we use two leading conceptual

modeling languages

UML and OPM

Model Quality Assessment

8

• Categories to evaluate students’ conceptual models:

– Model clarity and understandability (MCU) check if the model can be easily understood

– Model correctness (MCorr) evaluation of the model’s syntax, structure and behavior (logic) correctness.

– Model completeness (MComp) ensuring that the model contains all the requirements included in the scope .

– Model documentation (MDoc) appropriateness and contribution of the textual explanations related to the model to clarify the model’s constructing considerations

9

Peer & Meta Assessment

• Peer assessment is a method in which students are asked to evaluate each other's work.

• It helps students develop high order thinking skills including:– analyzing– generalizing– evaluating – critical thinking.

Meta assessment: Assessment of assessment

Evaluating the level of each student’s peer assessment

10

Peer & Meta Assessment

11

Research Goal

Overall research goalTo develop and assess a new approach for teaching and assessing undergraduate students in large PBL

systems engineering courses.

Current study goalTo develop and validate a supporting

Web-based tool for meta-assessment (assessment of peers’ assessment)

12

Research Method - Participants

About 130 students (half females) who participated in the course

Specification and Analysis of Information Systems at the Technion, Israel Institute of Technology.

The course objective is to familiarize 5th Semester Industrial Engineering and Information Systems Engineering undergraduate students with analysis, modeling, design, and assessment of systems in general and of information systems in particular.

13

Research Method - Setup

• PBL approach – reverse engineering of complex Web-based system, such as Gmail, Expedia, or eBay

– Realistic problems

– No single correct answer

• Groups of students were asked to specify, analyze

and model these complex system in both – OPM and

UML.

14

Research Method - Setup

• There were 23 groups of six (some of five) students divided into two teams of three (some of two).

• OPM and UML were taught every other week alternately.

GroupGroupGroupGroupGroupGroup

Team 1 Team 223

15

Research Method - Setup• crossover method –

– The first team started to model the system using OPM,while the second team started to model the same system using UML.

– In the middle of the semester, the teams swapped, and each team continue the model that the other team in the group had started.

G01 - Gmail

Team 1 Team 2

OPM

UML

UML

OPM

17

Research Method – Tools

Each student was tasked with assessing a two systems.

A dedicated Web-based peer assessment tool was developed

18

Research Method – Tools

Each student was tasked with assessing a two systems.

A dedicated Web-based peer assessment tool was developed

Students assessment was based on a list of criteria for each category.

19

Research Method - Tools

• The online assessment included:– a written assessment

– a corresponding grade in a scale of 1-10.

• To prevent students from giving 10 to all four models, the tool prevents giving any grade more than once to any one of the four models in each category

A dedicated Web-based peer assessment tool was developed

Research Method – Tools

• A dedicated Web-based meta-assessment tool was developed to evaluate the level of each peer assessment

• The inputs were:

− Student’s grading for each model

− Student’s explanations for the grading

− Model’s descriptive statistics based on other students grading

Research Method – Tools

• A dedicated Web-based meta-assessment tool was developed to evaluate the level of each peer assessment

• The inputs were:

− Student’s grading for each model

− Student’s explanations for the grading

− Model’s descriptive statistics based on other students grading

Research Method – Tools

• A dedicated Web-based meta-assessment tool was developed to evaluate the level of each peer assessment

• The inputs were:

− Student’s grading for each model

− Student’s explanations for the grading

− Model’s descriptive statistics based on other students grading

Research Method – Tools

• A dedicated Web-based meta-assessment tool was developed to evaluate the level of each peer assessment

• The outputs are:

− Grade for each categorywas given to the student based on his grades & explanation to all 4 models

− Explanation for the grading

Research Method – Tools

• A dedicated Web-based meta-assessment tool was developed to evaluate the level of each peer assessment

• The outputs are:

− Grade for each categorywas given to the student based on his grades & explanation to all 4 models

− Explanation for the grading

Research Method – Tools

• A dedicated Web-based meta-assessment tool was developed to evaluate the level of each peer assessment

• Evaluating scale:

−A list of available gradesand compatible comments.

26

Research Method – ToolsMeta-assessment grading scale • We created a grading scale

based on themes that had emerged from students' explanation combined with Bloom's revised taxonomy.

Research Method – ToolsMeta-assessment grading scale

Evaluation Evaluation definition Examples

Satisfactory = 60 Basic understanding – several basic summarized criteria with some diagram examples.

On overall the work is nice but some arrows and links cross each other (SD0, SD1) and makes it hard to read as well as the SD1.1 diagram included more than 5 processes and it is over loaded.

Good (-)= 75 Good understanding – several criteria with some diagram examples – mismatch between the explanation and grading.

Sometimes it's hard to read the diagrams because arrows cross each other (fig. 1.3, 1.5) and the caption on the lines is too small (fig. 1.1, 1.2). Additionally, some diagrams include unused objects (fig 1.4, 1.5)- adding unnecessary information can create confusion

• Three different aspects with examples that reinforce the main claim. •Not clear why this model was graded as 10?

Research Method – ToolsMeta-assessment grading scale

Evaluation Evaluation definition ExamplesExcellent = 100 Complex understanding – high

analyzing ability – several criteria with diagram examples – Including recommendations for improvement.

There is unfolding mixed with in zooming (Fig 2.3, 2.4), For better understanding- it should be separate.

… maybe I would use more tagged structural links for better explanation (Fig.6).

Complex understanding –ability to synthesis & conclude – several criteria with several different types of diagrams examples – drawing conclusions based on criteria (findings meaning).

A few mistakes that don't seem to indicate a big lack of understanding (one object is not connected to a process, XOR link missing-Fig.6), overall- the model seems to be correct.

Complex understanding – ability to evaluate & compare - several criteria with several different type of diagram examples - drawing conclusions based on comparing to the other works (grading explanation).

… (Fig 2.5,2.7). Overall- more mistakes then the other OPM team, BUT their diagrams are more detailed and informative, that uses correct states of object's and tagged structural links which makes it better in my opinion.

•Finding generalization with examples and conclusions•Critical thinking.

30

Research Method – ToolsMeta-assessment grading scale • We created a grading scale

based on themes that had emerged from students' explanation combined with Bloom's revised taxonomy.

• Reliability of the grading scale was tested by five judges.

• They were asked to evaluate several identical answers based on the grading scale.

Correlation between judges ranged from .78 to .93 and were significant (p<.01).

Research QuestionsFor validating the meta-assessment tool we asked four questions:

1 Is there a significant correlation between the students' meta-assessment grades and their prerequisite grades in software engineering course?

2 Is there any effect of the student's level of involvement in teamwork on student's meta-assessment grade?

3 Is there any gender effect on students' meta-assessment grades?

4 Is there any learning order effect on students' meta-assessment grades?

Criterion Validity

construct validity by group comparison

absence of bias

absence of bias

32

Results

The two grade variables were significantly correlated

r(128)=.69, p<.01.

This finding indicates empirical validity by criterion

The 1st research question was tested by a Pearson correlation test

Criterion Validity

33

Results

Main Effect n mean (sd) F(1,123) p

Teamwork involvement level

33% or lower 56 65.02 (18.06)18.65 P<.001

Greater than 33% 75 77.99 (16.15)

GenderMale 66 73.33 (19.02)

1.48 n.s.Female 65 71.54 (17.23)

Learning order (first modeling language)

UML 65 70.77 (18.77).22 n.s.

OPM 66 74.09 (17.41)

Meta assessment grade - means, standard deviation and the three way ANOVA main effect results

The two to four research questions were tested bythree-way ANOVA test

construct validity by group comparison

34

Results

Main Effect n mean (sd) F(1,123) p

Teamwork involvement level

33% or lower 56 65.02 (18.06)18.65 P<.001

Greater than 33% 75 77.99 (16.15)

GenderMale 66 73.33 (19.02)

1.48 n.s.Female 65 71.54 (17.23)

Learning order (first modeling language)

UML 65 70.77 (18.77).22 n.s.

OPM 66 74.09 (17.41)

Meta assessment grade - means, standard deviation and the three way ANOVA main effect results

The two to four research questions were tested bythree-way ANOVA test

construct validity by group comparison

absence of bias

35

Results

Main Effect n mean (sd) F(1,123) p

Teamwork involvement level

33% or lower 56 65.02 (18.06)18.65 P<.001

Greater than 33% 75 77.99 (16.15)

GenderMale 66 73.33 (19.02)

1.48 n.s.Female 65 71.54 (17.23)

Learning order (first modeling language)

UML 65 70.77 (18.77).22 n.s.

OPM 66 74.09 (17.41)

Meta assessment grade - means, standard deviation and the three way ANOVA main effect results

The two to four research questions were tested bythree-way ANOVA test

absence of bias

absence of bias

construct validity by group comparison

36

Conclusions

We developed and validated the meta-assessment tool

Content validation• three experts • correlation with other outcomes

structure validation• correlation between five judges (reliability of the scale)• group comparison (construct validity)

Meta assessment

tool

Absence of bias (reliability of the tool)• by gender• by order of using the modeling languages

37

Discussion• Meta assessment which is an extension for peer assessment

was found to be reliable and valid.

• Peer & meta assessment method enables to:– evaluate the individuals that take part in executing a team-based project. – promotes students' application of higher order thinking skills and

development of their metacognitive skills such as:

• Planning the execution of the task

• Allocating the optimal amount of time

• Monitoring and evaluating strategies

• Reflecting their project constructing

• Critical thinking

• Analyzing

• Synthesizing

• Evaluating the quality of each model

• Comparing between models

38

Further Study

We call for additional studies to:

– Validate the assessment tool in additional courses and countries

– Use NLP (natural language processing) methods to analyze large amount of peer assessments.

39

ACKNOWLEDGEMENT

This work was funded by:

– the EU FP7 Project #262044-VISIONAIR-Vision Advanced Infrastructure for Research

– the Gordon Center for System Engineering at the Technion

– the Technion Post-Doctoral Fellowships