Evaluation of the Current Learning Management System at JMUThe Students’ Perspective on Blackboard

Presented By

Eric JacobsMichael MayJason PooleJessica Robinson

IntroductionReview of Evaluation Process

Background

IntroductionThis evaluation plan was prepared by Eric Jacobs, Michael May, Jason Poole, and Jessica Robinson who are graduate students in the Educational Technology department at James Madison University. Coordinating efforts by Andrea Adams of the Center for Instructional Technology (CIT), and by Dr. Michele Estes of the Educational Technology department at James Madison University have helped advise this evaluation plan.

Blackboard is a learning management system that has the ability to act as the central hub for the curriculum. This has become increasingly apparent as distance learning programs depend on the features of this evaluand to manage course content and interact with students in a web-based environment. Blackboard’s diffusion wide, claiming more than 1,600 institutions using their LMS (Blackboard, 2011).

Purpose of evaluationReview of Evaluation Process

Stakeholders

Purpose

• To determine the needs that students have for learning management system.

• To identify the technologies that faculty and students are currently using in teaching and learning

• To identify the technologies that faculty and students are no long using in teaching and learning.

• Primary Stakeholders• Andrea Adams• Dr. Michele Estes

• Secondary Stakeholders• Evaluation team• Students of JMU

Purpose of evaluationReview of Evaluation Process

Decisions to be addressed

• To what degree does Blackboard meet the expectations and learning needs of undergraduate and graduate students at JMU?

• Do graduate and undergraduate students feel that Blackboard is a viable solution as a learning management system for JMU over the next 5 years?

• To what degree do the features in Blackboard meet the technological needs of the students at JMU?

Evaluation questionsReview of Evaluation Process

Needs related

What are students needs in regards to using a learning management system in their coursework?

What expectations do students have about learning management systems?

What technologies are students using both in Blackboard and in their day to day lives?

Do students feel that the technology in Blackboard is beneficial to their learning?

What features of Blackboard do students find obsolete or in need of updating?

What features in Blackboard do students engage with most often?

What features in Blackboard do students feel are the most beneficial?

Technology related

Features related

Methods of evaluationReview of Evaluation Process

Methods

• Cross-sectional Survey• Census• Interview

The SampleReview of Evaluation Process

• 2 undergraduate• 2 graduate• Purposive sample of Ed Tech Students

• ~20,000 students• Expected return rate ~2%

Sample for interviews

Sample for census survey

• Students in high Blackboard use courses

• 2,000 students or 10% of JMU enrollment

• Expected return rate 50%

Sample for census survey

InstrumentsReview of Evaluation Process

• Created and administered using Qualtrics.

• Recorded using Elluminate Live!

Surveys

Interviews

Evaluation planReview of Evaluation Process

Limitations

• Time Constraints• Return rate of survey data• Proximity of group members from JMU and each other

Logistics & Timeline

Budget

• Interviews and survey data completed in November

InterviewSummary #1

Interviewee DataGender: Female

Program of Study: Communications Major, Educational Media Minor

Class: Junior

General Info: Student has taken hybrid courses and is currently enrolled in an online course. Experience using Blackboard in high school.

+-

Positives

Negatives

• Access to grades• The more information on Blackboard

the better• E-mails of announcements• Organization of content• Discussion boards

• Announcements when viewed on Blackboard

• Instructors that don’t update Blackboard beyond syllabus

InterviewSummary #2

Interviewee DataGender: Female

Program of Study: Educational Technology

Class: Graduate

General Info: All classes were taken on Blackboard.

+-

Positives

Negatives

• Internal Blackboard email tools• Access to everything in central location

• Dislike wiki feature

InterviewSummary #3

Interviewee DataGender: Female

Program of Study: Middle School Education

Class: Junior

General Info: Overall positive experience with Blackboard

+-

Positives

Negatives

• All important documents in one place• Online grade book• Roster for contacting classmates

• Lots of features she doesn’t know how to use

• Inconsistency among JMU faculty

InterviewSummary #4

Interviewee DataGender: Female

Program of Study: Educational Technology

Class: Graduate

General Info: Overall positive experience using Blackboard. Almost all courses using Blackboard.

+-

Positives

Negatives

• Blackboard and Elluminate are perfect coupling

• Greater connection with course and instructor

• E-mail features, discussion board, grade feature

• Links and Resources

• Instructor implementation• Instructor Usage• Collaboration -- Real-time lacking• Group sharing is “clunky”• Synchronous info sharing (IM, chat)

Gender

Census Survey Data

# Answer Response %

1 Male 57 27%

2 Female 151 73%

Total 208 100%

1. Please select one: Male or Female

EducationalStanding

Census Survey Data

# Answer Response %

1 Freshman 51 25%

2 Sophomore 27 13%

3 Junior 41 20%

4 Senior 48 23%

5 Graduate Student 41 20%

Total 208 100%

2. Please select one: Specific Educational Standing

Census SurveyData

Census Survey Data

# Answer Response %

1 College of Buisness 25 12%

2 College of Communications 14 7%

3 College of Education 29 14%

4 College of Health and Human Services 31 15%

5 College of Humanities and Social Sciences

44 21%

6 College of Science, Technology, Engineering and Mathematics

47 23%

7 College of Visual and Performing Arts 8 4%

8 Undecided 10 5%

Total 208 100%

3. Please select one: Program of Study

Course interactions with BlackboardCensus Survey Data

# Answer Response %

1 I have taken a fully online course that used Blackboard

57 27%

2 I have taken a hybrid/blended course that meets online and in the classroom that used Blackboard

54 26%

3 I have taken face-to-face classroom courses that utilized Blackboard

191 92%

4 I have taken face-to-face classroom courses that do not utilize Blackboard

125 60%

4. Based on your course work at JMU, please describe your course interactions with Blackboard.

CommunicationToolsCensus Survey Data

# Answer Response %

1 social networks 177 85%

2 text messaging 196 94%

3 instant messaging 53 25%

4 email 200 96%

5 other 19 9%

5. Which technologies play a role in your daily communications?

Regularly used technologies

Census Survey Data

# Answer Response %

1 smartphone 109 55%

2 tablet 14 7%

3 laptop/desktop 194 98%

4 Other 29 15%

6. What technologies do you regularly use?

Would like social features in Blackboard

Census Survey Data

# Answer Response %

1 yes 64 32%

2 no 57 29%

3 unsure 76 39%

Total 197 100%

7. Would you like to see Blackboard have more social features for learning?

Blackboard fits expectationsCensus Survey Data

# Answer Response %

1 Strongly Disagree 4 2%

2 Disagree 11 6%

3 Neither Agree nor Disagree

46 23%

4 Agree 114 58%

5 Strongly Agree 22 11%

Total 197 100%

8. Blackboard fits my expectations of how a learning management system should work.

Overall experience with BlackboardCensus Survey Data

# Answer Response %

1 negative 3 2%

2 somewhat negative 15 8%

3 neutral 30 15%

4 somewhat positive 89 45%

5 positive 60 30%

Total 197 100%

9. Overall, how would you describe your experience using Blackboard?

How difficult is Blackboard

Census Survey Data

# Answer Response %

1 Very Difficult 2 1%

2 Somewhat Difficult 20 10%

3 Somewhat Easy 113 57%

4 Very Easy 62 31%

Total 197 100%

10. Overall, how difficult is Blackboard to use?

Satisfaction with Blackboard reliabilityCensus Survey Data

# Answer Response %

1 Very Dissatisfied 2 1%

2 Dissatisfied 13 7%

3 Neutral 33 17%

4 Satisfied 122 62%

5 Very Satisfied 26 13%

Total 196 100%

11. Please describe your satisfaction level of Blackboard as a reliable service.

Difficulty locating info in BlackboardCensus Survey Data

# Answer Response %

1 Very Difficult 2 1%

2 Difficult 24 12%

3 Neutral 62 32%

4 Easy 90 46%

5 Very Easy 18 9%

Total 196 100%

12. Please describe the difficulty in locating information within Blackboard.

How often do you access BlackboardCensus Survey Data

# Answer Response %

1 Never 0 0%

2 Less than Once a Month 0 0%

3 Once a Month 3 2%

4 2-3 Times a Month 3 2%

5 Once a Week 11 6%

6 2-3 Times a Week 72 37%

7 Daily 107 55%

Total 196 100%

13. How often do you access Blackboard?

Importance of features

Census Survey Data

# Question Not Very Important

Not Important

Important Very Important

Responses Mean

1 Email 16 13 80 78 187 3.18

2 Course Annoucements

2 5 64 116 187 3.57

3 Social Media

51 86 42 8 187 2.04

14. Please indicate the importance of the listed features for a learning management system

Importance of features

Census Survey Data

# Question Not Very Important

Not Important

Important Very Important

Responses Mean

1 Discussion Boards

16 33 94 44 187 2.89

2 Blogs 34 101 47 5 187 2.12

3 Wiki 41 96 42 8 187 2.09

15. Please indicate the importance of the listed features for a learning management system

Importance of features

Census Survey Data

16. Please indicate the importance of the listed features for a learning management system

# Question Not Very Important

Not Important

Important Very Important

Responses Mean

1 My Grades 2 1 22 162 187 3.84

2 Test 5 18 58 106 187 3.42

3 Rubrics and Learning Outcomes

1 10 59 117 187 3.56

Importance of features

Census Survey Data

# Question Not Very Important

Not Important

Important Very Important

Responses Mean

1 Assignments

2 1 41 143 187 3.74

2 Calendar 6 33 72 76 187 3.17

3 Groups 13 52 90 32 187 2.75

4 Journals 26 79 57 25 187 2.43

5 Learning Module

16 61 76 34 187 2.68

6 Survey 22 72 66 27 187 2.52

7 Podcasts 36 83 52 15 186 2.25

17. Please indicate the importance of the listed features for a learning management system

Most often accessed featuresCensus Survey Data

18. Which features do you access MOST often in Blackboard?

Most often accessed featuresCensus Survey Data

18. Which features do you access MOST often in Blackboard?

Least often accessed featuresCensus Survey Data

19. Which features do you access LEAST often in Blackboard?

Least often accessed featuresCensus Survey Data

19. Which features do you access LEAST often in Blackboard?

Importance of features

Census Survey Data

20. Blackboard meets my needs in my current courses.

# Answer Response %

1 Strongly Disagree 6 3%

2 Disagree 13 7%

3 Agree 108 58%

4 Strongly Agree 59 32%

Total 186 100%

Comments

Census Survey Data

Open comments section

• Received 35 comments

• 16 comments directly related to instructor knowledge and implementation of Blackboard

• 8 comments related to amount of tools available vs. the few tools used.

• Other less frequent comments• Smartphone support• Interface or navigation confusion• Dislike changes from previous

versions

Comments

Census Survey Data

Example comments

“I think that blackboard does not need to be overloaded with features. From my experience, most of the features in blackboard are not used because they are not very important for a classroom setting. Blackboard is useful because it allows an instructor to communicate with his or her students and students to communicate with their teacher. There is no need to improve blackboard or add more features to it because it already does what it does well. In fact, I find that having too many features makes programs stressful and difficult to use because they demand too much of users. I don't want a comprehensive life planner and entertainment center with blackboard. It already does what I ask of it: It provides a way for my teachers to communicate with me about upcoming assignments and grades. It does this very well.”

Comments

Census Survey Data

Example comments

“In my experience, use of blackboard as a learning management tool largely depends upon the instructor. Only a few of my classes have used discussion board and only one has used Elluminate Live! For most classes, blackboard serves as a way to distribute class documents, announcements, and contact information. Therefore, most of my use of blackboard has been with simple tools, which is really all I need blackboard for.”

“Some features are redundant and confusing, such as Journals and Online Journal. There is a lack of uniformity between how different professors use different tools. It's very clunky and clumsy. I wish it had a more seamless user interface, like facebook.”

Conclusions &Recommendations

Recommendations

Conclusions

• Students are overall pleased with Blackboard as a tool.

• Many of Blackboards features are unused in courses.

• Students feel instructors us of Blackboard is primary to its success in courses.

Option 1Keep Blackboard as JMU’s learning management system and focus attention on course implementation and faculty training.

Option 2Investigate alternative tools that better implement the functions most used by instructors in their courses.

Reflections of evaluation team +

-Positives

Negatives

• Recording interviews using Elluminate

• Data collection using Qualtrics

• Mass student email process

• Timing of survey

• Evaluators distance from JMU campus