Evaluation Report For FDP Design Activity on Active ...€¦ · Evaluation Report For FDP Design...

Team 117 – Self Evaluation Report

1

Evaluation Report

For

FDP Design Activity on

Active Learning - Peer Instruction

and Think-Pair-Share

Team Id : 117

Team Details:

Team Leader : Dr.D.Jeya Mala

Team Member 1: Dr.Harun Patel

Team Member 2: Mr.Sanju Mahawar

Team Member 3: Dr.Ranjan Kumar Mandal


2

TABLE OF CONTENTS

I. ANALYSIS OF ENGAGEMENT 3

A) What was the gross engagement in the course? [Macro level] 3

B) What was the engagement pattern of individual LeDs, LbDs, LxIs, and LxTs?

[Micro Level] 5

C) What is the transition pattern of engagement of learners across LeD, LbD, LxI,

and LxT? 7

II. ANALYSIS OF LEARNING 8

A) What is the gross learning in the course?[Macro] 8

B) What is the learning pattern across LbDs and Quizzes? [Micro] 10

C) What is the transition pattern of learning across LbDs and Quizzes? [Meso

Level] 11

III. ANALYSIS OF PERCEPTION 12

What was the perception of learners about the course? [Macro] 12

REFLECTIONS ON ANALYSIS 18

What do gross data about learning, perception and engagement tell about the

course? 18

What does the analysis tell about the course experience? 18

What needs improvement in the current course offering? 18

IV. WORKLOAD DISTRIBUTION 20


3

I. ANALYSIS OF ENGAGEMENT

A) What was the gross engagement in the course? [Macro level]

The MOODLE course on Active Learning – Peer Instruction and Think-Pair-Share was originally

designed as part of a blended course which was to be taken by participants of IITBx – FDP301x -

Mentoring Educators in Educational Technology. There were 7 peer reviewers who have evaluated this

course and 4 teachers along with 14 colleagues and so a total of 25 participants have registered for this

course.

The course involved the following activities that were to be completed by participants during the duration

of the course:

LeDs on Active Learning – 1 No.

LeDs on PI – 3 Nos

LeDs on TPS - 2 Nos

LbDs on Active Learning , PI and TPS- 6 Nos

Discussion on Active Learning , PI and TPS - 3

Reflection Quiz (RQ)

Advancing Your Learning (Lxt) - 8 Videos and 3 Resources

Assimilation Quiz (AQ)

Knowledge Quiz (KQ)

Resource Creation Assignment (RCA) - Designing your own PI and TPS

Peer review of RCAs

Evaluation and Feedback

During this FDP design activity, as a team we have discussed a lot on how to develop it as an effective

tool for the learners to practice Active Learning in their classroom by implementing Peer Instruction and

Think-Pair-Share (TPS) activities.

The team has applied TPACK model and Kirkpatrick's Training Evaluation Model throughout the

design of the course.

The activity completion report generated from the course Reports section shows the engagement of

participants in the activities as shown in Table 1.]

Table 1: Activity Completion Summary


4

Activity Completion Status

LeDs 1 25

2 25

3 25

4 25

5 25

6 25

LbDs 1 16

2 18

3 17

4 16

5 16

6 16

Discussion on LxT 16

Discussion on LxI 15

RQ 19

LxTs 11

AQ 17

KQ 19

RCA - Design your own PI and TPS -

Peer Review PI and TPS -


5

B) What was the engagement pattern of individual LeDs, LbDs, LxIs, and

LxTs? [Micro Level]

Looking into the performance and engagement of individuals in the course activities, following is the

engagement pattern of selected participants (those who have peer reviewed our course design) in the

course. Seven participants were selected for the analysis so that we have a sample groups showing high

engagement, medium engagement and low engagement pattern. Table 2 shows the level of engagement

of the seven sampled participants in the course.

Table 2: Participation of individual learners in the course

Participant

1

Participant

2

Participant

3

Participant

4

Participant

5

Participant

6

Participant

7

LeDs 1 Yes Yes Yes Yes Yes Yes Yes

2 Yes Yes Yes Yes Yes Yes Yes





LbDs 1 Yes Yes Yes Yes Yes Yes Yes






Discussion

on PI

Yes No Yes No Yes No Yes


6

Discussion

on TPS

No No No Yes No Yes No

RQ Yes No Yes Yes Yes Yes Yes

LxTs No No Yes No Yes No Yes

AQ Yes No Yes Yes Yes Yes Yes

KQ Yes No Yes Yes Yes Yes Yes

RCA -

Design

your own

PI and TPS

-- -- -- -- -- -- --

Peer

Review

RCA - PI

& TPS

-- -- -- -- -- -- --

From table 2, it can be inferred that, the sample participants taken from the list have attended the

initial set of activities to attend the LbDs based on LeDs. But then, the extended resources such

as LxTs in terms of videos and research papers have been given less importance and many of the

listeners have not viewed them.

Similarly, the sample data shows that, the participants are posting their replies to a single

discussion forum topic only and not the next one. This seems that, the discussion forums must be

restricted to one for any of the topic and several threads should be created from that topic.


7

C) What is the transition pattern of engagement of learners across LeD,

LbD, LxI, and LxT?


8

II. ANALYSIS OF LEARNING

A) What is the overall learning in the course “Peer Instruction &

Think Pair Share” at macro level?

Evaluation of the learning of the participants can be known by evaluating the marks

scored by the participants in attempting Knowledge quiz (KQ), Assimilation quiz (AQ),

Reflection quiz (RQ) and LbDs activities. Team were incorporated eight quizzes including

KQ,AQ,RQ and LbDs. And weighted score for each activity as follows:

Learning by Doing (LbD) Activities [50 marks]

Reflection Quiz [10 marks]

Assimilation Quiz [10 marks]

Knowledge Quiz [10 marks]

Peer Instruction Design Activity [20 marks]

Total 100 marks


9

Table 1: No. of participant’s attempted, average marks, maximum marks and average percentage

of participants who attempted each activity and their score.

LbDs KQ AQ RQ Activity Any All

No of

Participants

Attempted

15 16 16 15 0 11 8

Average

Marks

24.26 3.68 4.25 3.92

0 72 55.23

Maximum

Marks

50 10 10 10 20 100 100

Average

Percentage

48.52

36.8 42.5 39.2 - 72 55.23


10

B) What are the patterns of marking to learners across LbDs and

Quizzes?

Table 2: No. of participant’s attempted, average marks, maximum marks and average percentage

of participants who attempted each activity and total is calculated in 50 as it was the weightage

given to the LbDs in the whole course scoring.

LbD 1 LbD 2 LbD 3 LbD 4 LbD 5 Any of the

LbDs

All of the

LbDs

No of

Participants

Attempted

18 16 16 14 15 4 14

Average

Marks

7.88 6.28 8.28 8.28 7.73 13.12 39.3

Maximum

Marks

10 10 10 10 10 50

(Weighted)

50

(Weighted)

Average

Percentage

78.8 62.8 82.8 82.8 77.3 26.24 78.6


11

c) What is the transition structure of learning across LbDs and Quizzes?


12

III. ANALYSIS OF PERCEPTION

What was the perception of learners about the course? [Macro]

Analysis of perception of the designed FDP has been done through the feedback of the

participant in the three surveys:

1) Perception of Learning

2) Perception of Engagement

3) Perception of Usability

The detail of survey and responses are given below

1) Perception of Learning

Total 15 candidates have given their feedback on survey of “Perception of Learning”.

Table 3.1. Responses for the perception of learning

Questions Asked Strongly

Disagree

Mildly

Disagree

Neutral Strongly

Agree

Mildly

Agree

I understood the learning

objectives 0 1 2 9 3

I was able to relate each

learning objectives to the

learning I achieved

0 4 2 7 2

I was appropriately

challenged by the course

material

0 0 1 7 7

My learning was enhanced

due to the course 0 1 2 7 5


13

Figure 3.1. Bar diagram for the perception of learning

From the obtained responses, it can be seen that among the 15 participant, maximum participants

are strongly agree about perception of learning. 04 participants were saying that they were mildly

disagreeing in “relating the learning objectives to the learning they have achieved”. This result

suggests that there is need of further slender improvement in the course content.

2) Perception of Engagement

Total 12 candidates have given their feedback on survey of “Perception of Engagement”

Table 3.2. Responses for the Perception of Engagement


Disagree

Mildly

Disagree

Neutral Strongly

Agree

Mildly

Agree

I was well engaged during the

course 0 0 2 7 3

It was easy for me to get actively

involved each time I accessed the

course

0 0 1 5 6

I was given ample opportunity to

practice the skills I was to learn 0 1 1 5 5

I found quizzes/activities 1 1 2 3 5


14

interesting

I found the discussion forum

threads motivating enough to

participate regularly

1 1 2 4 4

I viewed/read all the additional

resources/reference materials. 0 1 2 7 2

Figure 3.2. Bar diagram for the perception of engagement

Figure 3.3. Bar diagram for the perception of engagement

From the feedback of participants (Bar Diagram) about the perception of engagement, we

concluded that all the participants were well engaged actively involved during the course and

they also read and viewed the additional resource/reference material given in the course. One


15

participant is strongly disagreeing in question asked related to the quizzes and discussion forum,

it indicate that there is need of improvement to make the quizzes and discussion thread more

interesting.

2) Perception of Usability

Total 11 candidates have participated in the feedback of perception of usability

Table 3.3. Responses for the Perception of Usability


Disagree

Mildly

Disagree

Neutral Strongly

Agree

Mildly

Agree

I found the course materials easy to

navigate 1 2 1 5 2

I was comfortable with the pace of the

program 1 2 1 4 3

I was comfortable with the duration of

the session 2 4 1 3 1

I found consistency in the course

material design 2 0 2 5 2

I found the course material clear and

understandable 1 1 0 4 5

I was given ample opportunity to

share my suggestion/feedback 1 2 1 3 4

I was given ample opportunity to get

answers to my queries 1 1 2 5 2

I found the discussion forum threads

relevant 1 1 0 2 7

I think that I would like to frequently

use what I learnt. 0 1 2 4 4


16

Fig. 3.4 – Bar diagram for the perception of Usability

Fig. 3.5 – Bar diagram for the Perception of Usability

Fig. 3.6 – Bar diagram for the Perception of Usability


17

We obtained multiple responses for the perception of usability (bar diagram and questions asked

in the feedback for the perception of usability). Majority of participants are strongly agreed in all

the question asked to them about the usability of the course. Two participants are strongly

disagreeing about the duration of the programme and consistency in the course material. There is

need to reduce the time of FDP and also slight changes are needed in the consistency of the

course.


18

REFLECTIONS ON ANALYSIS

1. What do gross data about learning, perception and engagement tell

about the course?

The gross data about engagement shows that the number of participants completing the LbDs/quizzes and LxTs / LxIs and discussion forum activities have decreased as the course progressed.

Also, from the data on scores of participants, we can see that the scores of the participants are gradually fluctuating (3.68% in KQ, 3.92% in RQ and 4.25% in AQ), although the number of participants attempting the final Knowledge Quiz (KQ) has drastically fallen (64% of the total participants enrolled). This indicates that the perceived learning value of

the course needs improvement. From the participants' perception of the course, it can be observed that the pace of the course and consistency of course material design needs improvement.

2. What does the analysis tell about the course experience?

Some of the positive sides in this analysis are

Learning part is good as the LeDs are viewed by all

LbDs are mostly attended by the participants

At least one of the discussion forums have been attended

Some of the LxT resources have been viewed

The analysis shows that the course needs the following improvements

Users engagement

levels of usability

3. What needs improvement in the current course offering?

The following sections need considerable improvement:

Engagement:

Engagement in Discussion forums

Completion of LxT resource readings

Completion of Activities and Quizzes Usability

Consistency in course material design

Duration of the session Learning

Reference materials/additional resources

More samples for practice


19

From the above sections, the following inferences have been made:

The FDP design has a lot of good features from which the participants were able to apply

this active learning strategy by means of Peer Instruction and Think-Pair-Share in their regular

class room set-up.

Based on the observations, it has been identified that, the time to complete the course

should be reduced in terms of reduction of number of LeDs, LxTs, LxI discussion forums and

LbD attempts to complete the course.

Also, to encourage the participants to be more engaged more examples to be provided

based on their domain such as Computer Science, Pharmacy, Bio-Technology, Maths etc.

In order to improve the scoring of the participants in Quizzes, the LbDs design must be

improved.

As a summary, this FDP design activity provides a good learning experience to the

learners in spite of the above said improvements.


20

IV. WORKLOAD DISTRIBUTION

The team consisted of 4 participants. The workload is distributed among them as follows:

1. Member 1: (Team leader)

1. Preparing the workload distribution 2. LxTs, LxIs; 3. Discussion Forums;

4. Resource Creation Assignment1 (RCA 1) for Peer Instruction; 5. Self Enrollment in Moodle 6. Progress Bar creation 7. Uploaded LxT resources to Moodle.

8. FDP design analysis report: Analysis and reflection of Engagement 9. FDP design report: reviewing and compiling reflections by team members

2. Member 2

1. Course Design Layout

2. Moodle Administrator (Setup) 3. Course Entry Survey, Exit Survey; 4. Feedback Forms; 5. Resource Creation Assignment 2 (RCA 2) for TPS

6. Moodle Gradebook setup and configurations 7. Survey report on Perception of Engagement, Learning and Usability

3. Member 3

1. LbDs,

2. Assimilation Quiz 3. Reflection Quiz 4. Knowledge Quiz 5. FDP design analysis report: analysis and reflection of learning

4. Member 4

1. LeDs 2.1, 2.2, 3.1, 3.2; 2. Uploading of LeDs 3. Moodle Course Creation Video upload; 4. FDP design analysis report: analysis and reflection of perception

Evaluation Report For FDP Design Activity on Active ...€¦ · Evaluation Report For FDP Design...

Documents

Transcript of Evaluation Report For FDP Design Activity on Active ...€¦ · Evaluation Report For FDP Design...