STUDENT SATISFACTION WITH WEB-BASED EDUCATIONAL RESOURCES FOR COMPUTER & INFORMATION SCIENCE:...
-
Upload
kathleen-knight -
Category
Documents
-
view
215 -
download
0
Transcript of STUDENT SATISFACTION WITH WEB-BASED EDUCATIONAL RESOURCES FOR COMPUTER & INFORMATION SCIENCE:...
STUDENT SATISFACTION WITH STUDENT SATISFACTION WITH WEB-BASED EDUCATIONAL WEB-BASED EDUCATIONAL
RESOURCES FOR COMPUTER & RESOURCES FOR COMPUTER & INFORMATION SCIENCE: Teaching INFORMATION SCIENCE: Teaching
with Merlot Collectionwith Merlot Collection
Emrah Orhun, Ph.D.Emrah Orhun, Ph.D.
Troy State University MontgomeryTroy State University Montgomery
Alabama, USAAlabama, USA
OutlineOutline
IntroductionIntroduction Learning and Teaching Materials in The Learning and Teaching Materials in The
Merlot CollectionMerlot Collection Students’ Satisfaction with Merlot Materials Students’ Satisfaction with Merlot Materials
in CIS Classesin CIS Classes ConclusionsConclusions
IntroductionIntroduction
Difficulties in locating relevant and Difficulties in locating relevant and affordable materials of high qualityaffordable materials of high quality – World Lecture Hall at the University of Texas World Lecture Hall at the University of Texas
» 83 categories to browse and search 83 categories to browse and search utilities to locate materialsutilities to locate materials
– WebCT (2002) WebCT (2002) » 1,250 publisher course content packs in more 1,250 publisher course content packs in more
than 50 subject areas than 50 subject areas » customizable materials around which to build customizable materials around which to build
courses combined with the functionality of WebCT's courses combined with the functionality of WebCT's course management software. course management software.
Quality issue Quality issue
World Lecture HallWorld Lecture Hall materials materials – courses offered at accredited colleges and courses offered at accredited colleges and
universities around the world universities around the world WebCT packsWebCT packs
– editorial standards of the publisher and editorial standards of the publisher and author of the textauthor of the text
MERLOT (Multimedia Educational Resource MERLOT (Multimedia Educational Resource for Learning and Online Teaching)for Learning and Online Teaching)
Web-based open resource designed Web-based open resource designed primarily for faculty and students in higher primarily for faculty and students in higher education education
emphasizes the quality issue emphasizes the quality issue
MERLOT repositoryMERLOT repository
– Over 8,000Over 8,000 free learning materials and free learning materials and support materials that can be integrated support materials that can be integrated within the context of a larger course within the context of a larger course
Figure 1. Entries in the Applications sub/subcategory of Information Technology subcategory under Science and Technology category
Support resourcesSupport resources
– sample assignments sample assignments
– evaluations of the learning materialsevaluations of the learning materials
»discipline panel peer reviews discipline panel peer reviews
»individual user reviews individual user reviews
Peer Review ProcessPeer Review Process
Discipline communities Discipline communities Biology, business, chemistry, engineering, Biology, business, chemistry, engineering,
health sciences, history, information health sciences, history, information technology, mathematics, music, physics, technology, mathematics, music, physics, psychology, teacher education, and world psychology, teacher education, and world languages. languages.
Discipline teams Discipline teams (“MERLOT Editorial Boards”) (“MERLOT Editorial Boards”)
– selected by the MERLOT partner organizations selected by the MERLOT partner organizations in accordance with the MERLOT guidelines. in accordance with the MERLOT guidelines.
– develop evaluation standards, develop evaluation standards, – conduct peer reviews of learning materials conduct peer reviews of learning materials
within the discipline, and within the discipline, and – make connections with their professional make connections with their professional
organizationsorganizations
Peer review processPeer review process
Evaluated by at least two team members who Evaluated by at least two team members who have relevant expertise. have relevant expertise.
Review process stagesReview process stages– triagetriage review to identify worthy candidates review to identify worthy candidates – review of candidates review of candidates – composite review report sent back to the authors composite review report sent back to the authors
for feedback and permission to post the review on for feedback and permission to post the review on MERLOT. MERLOT.
– approved review is posted to the MERLOT Web approved review is posted to the MERLOT Web site by the discipline leader site by the discipline leader
The evaluation criteria: The evaluation criteria: –quality of content, quality of content,
–potential effectiveness as a potential effectiveness as a teaching tool, teaching tool,
–and ease of use. and ease of use.
Each dimension evaluated separately Each dimension evaluated separately (on a scale of 1-5 stars, with 5 being the (on a scale of 1-5 stars, with 5 being the highest), and an overall rating is highest), and an overall rating is computed computed
A review must average three stars in A review must average three stars in order to be posted to the site. order to be posted to the site.
Table 1 Growth of MERLOT materials, Table 1 Growth of MERLOT materials, members, and peer reviews (May 28, 2002)members, and peer reviews (May 28, 2002)
TotalTotal AddedAdded
during last during last 365 days365 days
AddedAdded
during last during last 30 days30 days
MaterialsMaterials 68636863 1991 1991 229229
MembersMembers 86138613 4142 4142 600600
Peer Peer ReviewsReviews
479479 361361 2020
Table 1 Growth of MERLOT materials, Table 1 Growth of MERLOT materials, members, and peer reviews members, and peer reviews
((February 27, 2003February 27, 2003))
TotalTotal AddedAdded
during last during last 365 days365 days
AddedAdded
during last during last 30 days30 days
MaterialsMaterials 83318331 23382338 284284
MembersMembers 1263712637 55275527 712712
Peer ReviewsPeer Reviews 778778 8686 22
Member Member CommentsComments
17901790 305305 1818
AssignmentsAssignments 322322 5959 11
Table 1 Growth of MERLOT materials, Table 1 Growth of MERLOT materials, members, and peer reviewsmembers, and peer reviews
for IT Categoryfor IT Category
TotalTotal AddedAdded
during last 365 during last 365 daysdays
AddedAdded
during last 30 during last 30 daysdays
MaterialsMaterials 559559 274274 2121
MembersMembers 524524 259259 3030
Peer ReviewsPeer Reviews 3636 33 00
Member Member CommentsComments
218218 9999 66
AssignmentsAssignments 1111 99 00
May 25, 2002May 25, 2002 September September 26, 200226, 2002
February 27, February 27, 20032003
MaterialsMaterials 68636863 73957395 83318331
% Peer % Peer ReviewsReviews
7.07.0 8.58.5 9.39.3
% Member % Member CommentsComments
21.221.2 20.820.8 21.421.4
% % AssignmentsAssignments
3.43.4 3.83.8 3.93.9
Ratio of Peer Reviews, Member Comments, and Ratio of Peer Reviews, Member Comments, and Assignments to MaterialsAssignments to Materials
ProblemProblem Percentage of reviewed materials lowPercentage of reviewed materials low
Potential SolutionPotential Solution Peer review by community of learnersPeer review by community of learners
CauseCause Editorial peer review process too Editorial peer review process too
slowslow
STUDENTS’ SATISFACTION WITH STUDENTS’ SATISFACTION WITH MERLOT MATERIALS IN CIS CLASSESMERLOT MATERIALS IN CIS CLASSES
Goal of StudyGoal of Study
To find out to what extent MERLOT To find out to what extent MERLOT provides relevant and quality materials for provides relevant and quality materials for the CIS coursesthe CIS courses..
QuestionsQuestions
Is the quality and quantity of learning materials in Is the quality and quantity of learning materials in the MERLOT collection sufficient to satisfy CIS the MERLOT collection sufficient to satisfy CIS students’ knowledge/learning needs in specific students’ knowledge/learning needs in specific courses?courses?
Is the MERLOT collection ‘usable’ by the Is the MERLOT collection ‘usable’ by the students?students?
Does the MERLOT collection induce the students Does the MERLOT collection induce the students to become members of learning communities?to become members of learning communities?
Can peer review by students substitute for peer Can peer review by students substitute for peer review by editorial boards?review by editorial boards?
Questions re-formulatedQuestions re-formulated
How do students rate their satisfaction with How do students rate their satisfaction with MERLOT collection?MERLOT collection?
How do students rate the learning materials How do students rate the learning materials as reviewers? Do students’ ratings of the as reviewers? Do students’ ratings of the learning materials differ from the ratings learning materials differ from the ratings given by Discipline Panel members?given by Discipline Panel members?
Teaching with the MERLOT Teaching with the MERLOT Collection in CIS CoursesCollection in CIS Courses
Summer-Fall 2002 semesters in the following Summer-Fall 2002 semesters in the following CIS classes at Troy State University CIS classes at Troy State University Montgomery:Montgomery: – CIS 5547 CARCIS 5547 CAR Applied Systems AnalysisApplied Systems Analysis– CIS 5547 CBRCIS 5547 CBR Applied Systems Analysis Applied Systems Analysis – CIS 3332 CIS 3332 Software Engineering ISoftware Engineering I– CIS 3357 CIS 3357 Logical Structures of Logical Structures of
Computer DesignComputer Design
Overall, 41 students participated in the evaluation. Overall, 41 students participated in the evaluation. A total of 252 evaluations were made. A total of 252 evaluations were made.
Evaluation of web-based learning materials 1. 1. Visit MERLOT (www.merlot.org) site and register as a member.2. 2. For each chapter covered in the course:3. a. Find a relevant learning material in the IT subcategory.4. b. Review the material using the guidelines provided by MERLOT.
In the table provided below, state what learning outcomes are supported by the material and post your review as a Member Comment on MERLOT.
5. c. Rate your satisfaction with MERLOT collection on a scale 1-5, with 5 implying that you were fully satisfied with the quality and quantity of learning materials you found in connection with your task, and 1 implying that you were not satisfied at all.
6. d. Find a relevant online learning material not contained in MERLOT, post it to MERLOT and add a Member Comment.
Student Assignment
ChapterChapter TopicTopic TitleTitle URLURL Learning Learning outcomeoutcome
Satisfaction Satisfaction with MERLOT with MERLOT
collection for this collection for this task 5= very good task 5= very good
1=very poor1=very poor
11 Players in the Players in the System GameSystem Game
Enterprise Enterprise Resource Resource Planning Planning SupersiteSupersite
http://www.http://www.erpsupersiteerpsupersite.com.com
Provides Provides analysis on analysis on ERP trendsERP trends
33
22 Information Information System Building System Building BlocksBlocks
Interface Interface Design for Design for Educational Educational MultimediaMultimedia
http://www.http://www.cdlcdl..eduedu..rachelrachel/PDF//PDF/intdesmmintdesmm..pdfpdf
Describes the Describes the principle of principle of interface interface designdesign
44
33 Information Information System System DevelopmentDevelopment
Risk analysis Risk analysis and and management management for Projects for Projects (RAMP)(RAMP)
http://www.http://www.rampriskramprisk.com/.com/frontindexfrontindex..htmhtm
Comprehensive Comprehensive outline of outline of RAMPRAMP
55
Example of Relevant MERLOT Material
ChapterChapter TopicTopic TitleTitle URLURL Learning Learning outcomeoutcome
11 Players in the Players in the System GameSystem Game
New New responsibilities responsibilities Fall Heavy on IT Fall Heavy on IT WorkersWorkers
www.www.informationweekinformationweek.com.com
Talks about the Talks about the new job new job responsibilities responsibilities of current and of current and upcoming IT upcoming IT employees.employees.
22 Information Information System Building System Building BlocksBlocks
TP Middleware TP Middleware keeping up with keeping up with Mainframe Mainframe RevivalRevival
www.www.adtmagadtmag.com.com
Explains the Explains the functionality of functionality of the product TP the product TP middleware on middleware on mainframe mainframe systems.systems.
33 Information Information System System DevelopmentDevelopment
Tips for a Tips for a successful successful Software Software Distribution Distribution Project.Project.
www.www.computerworldcomputerworld.com.com
Shares other Shares other companies companies success stories of success stories of in relation to in relation to Software Software Distribution Distribution Project.Project.
Example of New MERLOT material posted
How do students rate their satisfaction with How do students rate their satisfaction with MERLOT collection?MERLOT collection?
COURSE
Number of
Students
Number of
Topics
Number of
Evaluations
Average sdev
CIS3357 Logical Structures of Computer Design 8 7 49 3.8 1.3
CIS5547 (A+B) Applied Systems Analysis 22 14 165 3.8 0.8
CIS3332 Software Engineering I 11 4 38 3.8 1.0
Students’ mean satisfaction ratings for MERLOT collection
Ch
Topic
NAverag
e sdev Min MaxPercent rated 4-5
1 Binary Numbers & Codes 8 4.7 1.4 3 5 87.5
2 Digital Circuits 8 4.6 1.5 3 5 75.0
3 Combinational Circuits 8 4.3 1.9 1 5 62.5
4 Sequential Logic Systems 8 4.2 1.9 1 5 62.5
5 Registers & Counters 8 3.8 2.1 1 5 50.0
6 Memory & Programmable Logic 5 3.2 1.5 1 5 40.0
7 Register Transfer & Computer Operations 4 3.8 1.9 1 5 75.0
CIS3357 Students’ satisfaction with MERLOT collection
Ch
Topic
NAverag
e sdev Min MaxPercent rated 4-5
1 Process 13 4.2 0.8 1 5 76.9
2 Project Management 12 4.2 0.4 2 5 91.7
3 Requirements Analysis I 11 3.8 1.2 1 5 72.7
4 Requirements Analysis II 10 3.6 1.2 1 5 70.0
5 Software Architecture 13 4.2 0.8 1 5 76.9
6 Detailed Design 12 4.2 0.4 2 5 91.7
CIS3332 Students’ satisfaction with MERLOT collection
Ch
Topic
NAvera
ge sdev Min MaxPercent rated
4-5
1 Players in the System Game 22 3.6 0.8 2 5 59.1
2 Information System Building Blocks 20 3.7 0.8 2 5 55.0
3 Information System Development 19 3.6 1.2 1 5 63.2
4 Project Management 10 4.1 0.7 3 5 80.0
5 Systems Analysis 12 3.8 0.6 3 5 75.0
6 Requirements Discovery 12 3.9 0.8 3 5 66.7
11
Application Architecture and Modeling
10 3.9 0.7 3 5 70.0
CIS5547 Students’ satisfaction with MERLOT collection
A Object-Oriented Analysis and Modeling 9 3.7 0.7 3 5 55.6
B Object-Oriented Design and Modeling 9 3.9 0.8 3 5 66.7
13
Output Design and Prototyping 9 4.0 0.5 3 5 88.9
14
Input Design and Prototyping 9 4.2 0.4 4 5 100.0
15
User Interface Design 9 3.9 0.8 3 5 66.7
16
Systems Construction and Implementation
7 4.1 0.9 3 5 71.4
17
Systems Operations and Support 8 4.1 0.8 3 5 75.0
CIS5547 Students’ satisfaction with MERLOT collection (contd.)
How different are students’ How different are students’ evaluations from Editorial Peer evaluations from Editorial Peer
Reviews?Reviews?
TypeType ****** ******** **********
TutorialTutorial File System DesFile System Design for an NFS ign for an NFS File Server AppFile Server Applianceliance
Cascading StyleCascading Style Sheets, Level1 Sheets, Level1
Electronic ComElectronic Commerce Tutorial merce Tutorial
FTP TutorialFTP Tutorial
Learning Learning Microsoft Word Microsoft Word 20002000
Web-Database Web-Database IntegrationIntegration
Mulder's Mulder's Cascading Style Cascading Style Sheets TutorialSheets Tutorial
Sharky's Sharky's Netscape Netscape Frames TutorialFrames Tutorial
Tje ABCs of Tje ABCs of Building a Web Building a Web SiteSite
Data Data CommunicationCommunicationss
ADSLADSL
Computer Computer Knowledge Knowledge Virus Tutorial Virus Tutorial
Reference Reference MaterialMaterial
Study Guides Study Guides and Strategiesand Strategies
Tips for Using Tips for Using BlackboardBlackboard
AnimationAnimation TECHtionaryTECHtionary Warriors of the Warriors of the netnet
Distribution of materials in IT
Material Material TypeType
****** ******** **********
TutorialTutorial
ADSLADSL
Reference Reference MaterialMaterial
Web-DatabaseWeb-Database Integration Integration
AnimationAnimation TECHtionaryTECHtionary
Distribution of materials according to type and rating Distribution of materials according to type and rating (excluding materials with no user comments with 3 (excluding materials with no user comments with 3 categories)categories)
Sample of Materials from IT Category
Sample of StudentsSample of Students
CIS3332 SOFTWARE ENGINEERING (16)CIS3332 SOFTWARE ENGINEERING (16) CIS5547 SYSTEMS ANALYSIS (27)CIS5547 SYSTEMS ANALYSIS (27)
CaseCase Editorial Editorial Board Board RatingRating
Member Member RatingRating
Students’ Students’ RatingRating
(mean)(mean)TECHtionaryTECHtionary 3.03.0 ********** 3.753.75
Web-DatabaseWeb-Database Integration Integration 3.673.67 ****** 3.823.82
Students’ Ratings
MeanMean Std. Dev.Std. Dev. Std. ErrorStd. Error VarianceVariance Coef.Var.Coef.Var. CountCount
3.7543.754 .421.421 .086.086 .177.177 11.21811.218 2424
MinimumMinimum MaximumMaximum RangeRange SumSum Sum Sum SquaredSquared
# Missing# Missing
2.52.5 4.44.4 1.91.9 90.190.1 342.33342.33 00
Students’ ratings for TECHtionary
Editorial Board Rating: 3.00Editorial Board Rating: 3.00
MeanMean Std. Dev.Std. Dev. Std. ErrorStd. Error VarianceVariance Coef. Var.Coef. Var. CountCount
3.8293.829 .545.545 .119.119 .297.297 14.23814.238 2121
MinimumMinimum MaximumMaximum RangeRange SumSum Sum Sum SquaredSquared
# Missing# Missing
2.82.8 4.94.9 2.12.1 80.480.4 313.76313.76 00
Students’ ratings for Web-Database Integration
Editorial Board Rating: 3.67Editorial Board Rating: 3.67
Is the mean student rating different from the rating Is the mean student rating different from the rating given by the Editorial Board?given by the Editorial Board?
TECHtionary– Different (one-group t-value = 8.773, p=0.0001)
Web-Database Integration– Insufficient evidence (one-group t-value = 1.333,
p=0.1975)
ConclusionsConclusions
Students are satisfied with the Students are satisfied with the quality and quality and quantity of learning materials they found in quantity of learning materials they found in the IT category.the IT category.
Current data suggest that students rate Current data suggest that students rate materials differently than editorial boards.materials differently than editorial boards.
The present limited study does not provide The present limited study does not provide evidence for the notion that peer review by evidence for the notion that peer review by students can substitute for peer review by students can substitute for peer review by editorial boards.editorial boards.