Evaluation of Multimedia Evaluation of Multimedia Software and a Workbook Software and a Workbook

Designed to Improve 3-D Spatial Designed to Improve 3-D Spatial Skills of Engineering StudentsSkills of Engineering Students

Sheryl A. Sorby & Thomas DrummerSheryl A. Sorby & Thomas Drummer

Michigan Technological UniversityMichigan Technological University

BackgroundBackground

Spatial skills are important for success in Spatial skills are important for success in technical fieldstechnical fields Especially important for success in engineering Especially important for success in engineering

graphicsgraphics

Significant gender differences in 3-D spatial Significant gender differences in 3-D spatial skills, favoring malesskills, favoring males Gender differences cross international boundariesGender differences cross international boundaries

Poorly developed spatial skills may lead women Poorly developed spatial skills may lead women to drop out of engineering at a disproportionate to drop out of engineering at a disproportionate raterate

Assessing Spatial Skills-Purdue Spatial Assessing Spatial Skills-Purdue Spatial Visualization Test: Rotations (PSVT:R)Visualization Test: Rotations (PSVT:R)

Assessing Spatial Skills-Mental Assessing Spatial Skills-Mental Cutting Test (MCT)Cutting Test (MCT)

Assessing Spatial Skills-Differential Aptitude Assessing Spatial Skills-Differential Aptitude Test: Space Relations (DAT:SR)Test: Space Relations (DAT:SR)

Spatial Skills Class at Spatial Skills Class at Michigan TechMichigan Tech

Initially developed in 1993 with funding Initially developed in 1993 with funding from the National Science Foundationfrom the National Science Foundation

Additional funding in 1998 led to Additional funding in 1998 led to development of multimedia software and development of multimedia software and workbookworkbook

Class modified significantly in 2000 to Class modified significantly in 2000 to include software and workbookinclude software and workbook

Multimedia Software and Multimedia Software and WorkbookWorkbook

Developed in 1998-99 with funding from Developed in 1998-99 with funding from the National Science Foundationthe National Science Foundation

Consist of 9 separate modules designed to Consist of 9 separate modules designed to enhance 3-D spatial skillsenhance 3-D spatial skills Each module consists of Background and Each module consists of Background and

Exercises sectionsExercises sections Software is stand-alone and works on Software is stand-alone and works on

either a Mac or PCeither a Mac or PC Published by Delmar Learning in 2002Published by Delmar Learning in 2002

Assessment of Multimedia Assessment of Multimedia SoftwareSoftware

Students given attitudinal surveys rating Students given attitudinal surveys rating various aspects of the softwarevarious aspects of the software Ratings given by module and overallRatings given by module and overall

Ratings were on a 1-4 scaleRatings were on a 1-4 scale Students asked to rate such things as Students asked to rate such things as

ease of understanding, ease of use, etc.ease of understanding, ease of use, etc.

Results from Attitudinal Surveys of Results from Attitudinal Surveys of Individual ModulesIndividual Modules

ModuleModule BackgroundBackground ExercisesExercisesIsometric SketchesIsometric Sketches 3.523.52 3.353.35

Orthographic ProjectionOrthographic Projection 3.583.58 3.443.44

Flat Pattern FoldingFlat Pattern Folding 3.523.52 3.663.66

Rotation about One AxisRotation about One Axis 3.473.47 3.493.49

Rotation about Two AxesRotation about Two Axes 3.623.62 3.793.79

Reflections and SymmetryReflections and Symmetry 3.523.52 3.363.36

Cross Sections of SolidsCross Sections of Solids 3.633.63 3.583.58

Surfaces & Solids of RevolutionSurfaces & Solids of Revolution 3.653.65 3.543.54

Combining SolidsCombining Solids 3.503.50 3.533.53

Student Overall RatingsStudent Overall Ratings

StatementStatement AgreeAgree DisagreeDisagree UncertainUncertain

The modules helped me learn the The modules helped me learn the course materialcourse material

3232 11 22

The multimedia modules were fun to The multimedia modules were fun to useuse

3131 22 22

The topics covered in the modules were The topics covered in the modules were appropriate to the courseappropriate to the course

3535 00 00

The multimedia modules were easy to The multimedia modules were easy to useuse

2828 44 33

I would rather learn using the I would rather learn using the multimedia materials than using written multimedia materials than using written materialsmaterials

2121 55 99

Student Overall Ratings (cont.)Student Overall Ratings (cont.)

StatementStatement AgreeAgree DisagreeDisagree UncertainUncertain

The exercises in the multimedia modules The exercises in the multimedia modules were at the right difficulty level for mewere at the right difficulty level for me

1818 1717 00

There were enough exercises at the end There were enough exercises at the end of each module to find out whether or not I of each module to find out whether or not I understood the materialunderstood the material

2222 88 55

It took me the entire hour of lab time to It took me the entire hour of lab time to complete each modulecomplete each module

11 3131 33

This is my first time using multimedia This is my first time using multimedia educational softwareeducational software

2424 1010 11

Multimedia software is an effective tool for Multimedia software is an effective tool for helping students develop their 3-D spatial helping students develop their 3-D spatial abilityability

3232 11 22

Assessment of Gains on Spatial Assessment of Gains on Spatial Skills Test-PSVT:RSkills Test-PSVT:Rnn Pre-Pre-

TestTestPost-Post-TestTest

GainGain Significance Significance of Gainof Gain

Original Original Course Course (93-99)(93-99)

186186 50.550.5 76.976.9 26.426.4 p<0.0001p<0.0001

Modified Modified Course Course (00-02)(00-02)

157157 48.348.3 73.773.7 25.425.4 p<0.0001p<0.0001

Assessment of Gains on Spatial Assessment of Gains on Spatial Skills Test-MCTSkills Test-MCT

nn Pre-Pre-TestTest

Post-Post-TestTest

GainGain Significance Significance of Gainof Gain

Original Original Course Course (93-99)(93-99)

9999 37.937.9 51.451.4 13.513.5 p<0.0001p<0.0001

Modified Modified Course Course (00-02)(00-02)

109109 34.834.8 52.652.6 17.817.8 p<0.0001p<0.0001

Assessment of Gains on Spatial Assessment of Gains on Spatial Skills Test-DAT:SRSkills Test-DAT:SRnn Pre-Pre-

TestTestPost-Post-TestTest

GainGain Significance Significance of Gainof Gain

Original Original Course Course (93-99)(93-99)

9999 62.362.3 78.378.3 16.016.0 p<0.0001p<0.0001

Modified Modified Course Course (00-02)(00-02)

110110 62.162.1 77.977.9 15.815.8 p<0.0001p<0.0001

Improvement in Retention RatesImprovement in Retention Rates

WomenWomen MenMen

EGEG CGCG EGEG CGCG

Original Original CourseCourse

88.9%88.9% 68.3%68.3% 75.3%75.3% 69.0%69.0%

Modified Modified CourseCourse

87.4%87.4% 71.7%71.7% 76.8%76.8% 70.0%70.0%

Differences in Retention Rates for Women wereStatistically Significant but those for Men were not

Current ProjectCurrent Project

New course using multimedia software and New course using multimedia software and workbook seems to be effective workbook seems to be effective Similar or better gains on spatial skills testsSimilar or better gains on spatial skills tests Similar retention rates for men and women in Similar retention rates for men and women in

engineeringengineering

Research Questions: Is the software alone as Research Questions: Is the software alone as effective as when it is used in conjunction with effective as when it is used in conjunction with the workbook in developing spatial skills? and the workbook in developing spatial skills? and Can the materials be used effectively with non-Can the materials be used effectively with non-engineering students?engineering students?

Study GroupsStudy Groups

170 non-engineering students participated in the study170 non-engineering students participated in the study Most from either Biology or Computer ScienceMost from either Biology or Computer Science

Divided into four groupsDivided into four groups Software alone, Workbook alone, Software and Workbook, and Software alone, Workbook alone, Software and Workbook, and

ComparisonComparison Treatment groups participated in ten weeks of training in Treatment groups participated in ten weeks of training in

2-hour sessions each week2-hour sessions each week Tried to balance groups for gender, major, and PSVT:R Tried to balance groups for gender, major, and PSVT:R

pre-test scorepre-test score Comparison group invited back for post-test at end of 10-Comparison group invited back for post-test at end of 10-

week periodweek period

Gains on Spatial Skills Tests-Gains on Spatial Skills Tests-PSVT:RPSVT:R

Pre-testPre-test Post-testPost-test GainGain SignificanceSignificance

EG-SoftwareEG-Software 69.569.5 79.479.4 7.57.5 p<0.005p<0.005

EG-WorkbookEG-Workbook 75.975.9 86.686.6 11.611.6 p<0.0005p<0.0005

EG-Software & EG-Software & WorkbookWorkbook

74.374.3 85.685.6 11.211.2 p<0.0005p<0.0005

Comparison Comparison GroupGroup

72.672.6 77.977.9 5.35.3 p<0.03p<0.03

Gains on Spatial Skills Tests-MCTGains on Spatial Skills Tests-MCT

Pre-testPre-test Post-testPost-test GainGain SignificanceSignificance

EG-SoftwareEG-Software 44.544.5 51.051.0 6.56.5 p<0.01p<0.01

EG-WorkbookEG-Workbook 53.953.9 62.362.3 8.48.4 p<0.001p<0.001

EG-Software & EG-Software & WorkbookWorkbook

46.446.4 60.260.2 13.813.8 p<0.0005p<0.0005

Testing DataTesting Data

Each group had one or two data points that Each group had one or two data points that appeared to be outliersappeared to be outliers

For example in SW group, one foreign-born For example in SW group, one foreign-born student had pre-test of 17% and post-test of student had pre-test of 17% and post-test of 92%. 92%. Her English skills improved dramatically over 10-week Her English skills improved dramatically over 10-week

periodperiod Large gain could be due to improvements in language Large gain could be due to improvements in language

skills more than to improvements in spatial skillsskills more than to improvements in spatial skills Data analyzed using a nonparametric logistic Data analyzed using a nonparametric logistic

regression modelregression model

Logistic Regression ResultsLogistic Regression Results

Both groups that used the workbook were Both groups that used the workbook were significantly better than the CGsignificantly better than the CG

Software only group was not better than CGSoftware only group was not better than CG

Attitudinal Survey and ResultsAttitudinal Survey and Results

Students in EG were administered an Students in EG were administered an attitudinal survey at end of training periodattitudinal survey at end of training period

62% would have preferred using the 62% would have preferred using the software alone for training even though it software alone for training even though it was least effectivewas least effective Less time involvedLess time involved

Female students spent more time Female students spent more time completing the training materials for all completing the training materials for all groupsgroups

Attitudinal Survey and ResultsAttitudinal Survey and Results

Male students expressed greater confidence in Male students expressed greater confidence in their spatial skills at beginning of studytheir spatial skills at beginning of study Question asked retrospectivelyQuestion asked retrospectively

Level of ConfidenceLevel of Confidence Males (n=66)Males (n=66) Females (n=29)Females (n=29)

Not confidentNot confident 1.5%1.5% 27.6%27.6%

Somewhat ConfidentSomewhat Confident 28.8%28.8% 51.7%51.7%

ConfidentConfident 37.9%37.9% 20.7%20.7%

Most ConfidentMost Confident 31.8%31.8% 0%0%

Attitudinal Survey and ResultsAttitudinal Survey and Results Women preferred the use of manipulatives and working Women preferred the use of manipulatives and working

in groupsin groups Female students expressed higher level of learningFemale students expressed higher level of learning

Level of LearningLevel of Learning Males (n=66)Males (n=66) Females (n=29)Females (n=29)

Did not learn anything newDid not learn anything new 6%6% 3%3%

Learning limited to a few Learning limited to a few aspectsaspects

47%47% 17%17%

Learned something on most Learned something on most daysdays

26%26% 45%45%

Learned something every Learned something every sessionsession

21%21% 35%35%

ConclusionsConclusions

Multimedia software and workbook developed Multimedia software and workbook developed for use with engineering students is suitable for for use with engineering students is suitable for use with non-engineersuse with non-engineers

Software alone does not seem to be as effective Software alone does not seem to be as effective as when used in conjunction with the workbookas when used in conjunction with the workbook

There were gender differences in the type of There were gender differences in the type of training preferred and in the perceived level of training preferred and in the perceived level of learning learning

AcknowledgementAcknowledgement

The authors gratefully acknowledge the The authors gratefully acknowledge the support of this project from the National support of this project from the National Science Foundation through grant number Science Foundation through grant number HRD-0429020HRD-0429020