Running head: ONLINE LEARNING ASSESSMENT 1

Online Learning Assessment: Self- Regulation

Maimoona H. Al Abri

George Mason University

Spring 2016

EDRS 630

Running head: ONLINE LEARNING ASSESSMENT 2

TABLE OF CONTNET

ABSTRACT…………………………………………………………………………. 3

PROBLEM…………………………………………………………………………… 5

PURPOSE AND OBJECTIVES………………………………………………………. 5

REVIEW OF LITERATURE…………………………………………………………. 7

The IMPLICATION OF ONLINE LEARNING ASSESSMENT AND SELF-

REGULATION………………………………………………………………………..

10

THE DEVELOPMENT OF GOAL SETTING AND SELF ASSESSMENT IN SELF

REULATED LEARNING……………………………………………………………..

12

ONLINE LEARNING ASSESSMENT……………………………………………….. 15

ASSESSMENT INSTRUMNETS…………………………………………………….. 17

PROPOSED SOLUTION……………………………………………………………... 22

ASSESSMENT DEVELOPMENT…………………………………………………… 23

PILOT TESTING……………………………………………………………………… 34

REFLECTIVE DISCUSSION………………………………………………………… 61

REFERENCE…………………………………………………………………………. 63

Running head: ONLINE LEARNING ASSESSMENT 3

Abstract

Self-regulation has become a crucial factor in online learning. The distance learners is the owner

of the learning process and is responsible for the regulation and control of it. The present

assessment paper aims to assess the self-regulation skills of master teachers through technology-

based training (TBT) in the Specialized Center for Professional Training of Teachers (SCPTT).

This assessment paper includes a summative test which consists of ten multiple-choice questions

and a performance task. The construct was applied to 28 individuals who have taken online

courses. The participants responded to the test voluntarily through an online survey software

namely Survey Monkey. Online learning requires students to consider their learning pathway

and how to approach it. Therefore, I measured two components of self- regulation: goal setting

and self- assessment. I statistically interpreted the results of the test by analyzing the Central

Tendency and variability, validity and reliability of the test, and the items difficulty and

discrimination. Based on these tests the examiner evaluates whether the test was valid in order

to allow decision-makers to judge the participants’ performance and thereby, decides to make an

improvement in the processes, practices, and materials that are used at the SCPTT. Further

development of the test is required. Therefore, improvement will involve changing the type of

scale and focusing on three factors of self-regulation, planning, reflection, and monitoring.

Keywords: technology-based training, self-regulation, goal setting, self-assessment

Running head: ONLINE LEARNING ASSESSMENT 4

Online Learning Assessment

Problem

The World Bank report (2012) indicated that the decline of students’ learning performance in Oman,

Middle East, was regarded to less-skilled teaching methodologies. Simultaneously, Trends in International

Mathematics and Science Study (TIMMS) revealed that Omani students demonstrated low performance in

mathematics and science assessments for the fourth and eighth grades (Mullis & Michael, 2015). As a result, the

government has made it a priority to raise the quality of education in Oman. Integrating technology in education

plays a significant role in the refinement of teaching skills and knowledge into classrooms. Consequently, a

Specialized Center for Professional Development of Teachers (SCPTT) has been established approximately two

years ago, which intends to qualify Omani teachers through experiencing teaching practices. The professional

development in the SCPTT is delivering through: (a) face to face training using a wide range of active learning

methods; (b) online learning through individual and collaborative work through discussion forums and wiki; (c)

workplace learning. These processes, collectively, are provided through an e-learning platform. Moreover,

according to the feedback of trainers of the (SCPTT), they are encountered difficulties in assessing teachers

through conducting professional development via collaborative learning environment. The data analysis of this

study revealed that there are several evaluation strategies are used in SCPTT to assess the learning performance

of trainees during professional development. They are formative assessment and summative assessment with

different assessment tools. They including pre-assessment and post-assessment, self-assessment, individual

reflection on experience, task-based learning, a written descriptive report (a case study), and peer-evaluation in

some situations. However, the finding of the study revealed that the SCPTT lacks a clear criteria of assessment.

Most participants indicated that the lack of assessment criteria refers to the absence of rubrics and checklists to

guide trainers in evaluating trainees effectively.

ONLINE LEARNING ASSESSMENT 5

Consequently, it is necessary for the (SCPTT) to have a clear criteria for online learning assessment.

Furthermore, Miller and Tucker (20155) indicated that the challenges encountered educators through online

learning are the development of online learning pedagogy and assessment instruments. Similarly, Oncu and

Cakir (2011), stated that the assessment of online learning environment (OLE) is still an issue, and the critical

issue of it is assessing the quality of learning while the learners spend a long time on learning.

In addition, online learning is a way of learning and teaching relies on the constructivism Learning

Theory. Miller and Tucker (2015) clarified that the constructivist approach to online learning, assessment and

instruction, should concentrate on the role of learners or a group of learners. Furthermore, Dabbagh and

Bannan-Ritland (2005) described that online learning promotes self-directed learning, where the online learner

has a capability to define learning needs, set learning goals, define the resources, and evaluate learning

outcomes. They pointed that the self-directed learning can be acquired by the process of self-regulation

including, for instance, goal setting and self-evaluation. Consequently, online learning can be achieved by

considering the skills and techniques of self- regulation.

Purpose and Objectives

The purpose of the paper is to assess the self-regulation skills with master teachers through technology-

based training (TBT). Specifically, the assessment focused on the metacognitive components of self-regulated

learning. The self-regulation assessment measured two scales: (a) goal setting, and (b) self-assessment. The

overall goals of the assessment are to ensuring the quality of learning; developing standards of assessment for

online courses; and attaining the validity of the assessment in order to promote decision makers in the SCPTT to

improve the weakness area in the training materials.

In addition, the desire of developing the pilot test is based on my specialization of Instructional Design

and Technology, in which I am studying different pedagogical models of learning. Also, according to my

ONLINE LEARNING ASSESSMENT 6

personal qualitative study exploring trainers’ experiences in technology-based training, to ascertain what

meaning has been manifested as a result, what are the practices and learning activities embedded in the

instruction, and how this approach connects to the constructivism learning theories, I found that trainers in the

SCPTT are lacking for standards of assessment, specifically in online learning courses. In addition, because the

SCPTT is a new project in the MOE, there is a demand to ensure the quality of learning outcomes and the

validity of the assessment.

Moreover, self-regulated learning is an important factor that controls individuals’ learning at their own

pace, but it is a challenge in assessing those learners through online courses in terms of do they have the skills

and abilities to manage their own learning, to state their goals setting, and to be self-evaluated and reflect on

their experience in order to achieve the intended skills. Law, Ge, and Eseryel (2016) stated that there are lack

of instruments have measured SR into a collaborative context. Therefore, developing a plausible instrument to

measure to what extent learners have SR skills to regulate and control their own learning through learning

communities is required. In addition, the pilot test will gain me knowledge on the process of developing an

assessment for online courses. Also, it is beneficial to improve the assessment processes in the (SCPTT) as it is

a new project. Thus, support teachers to be self- regulated learner will enhance the success of technology-

based training in the SCPTT.

Review of Literature

Theoretical Framework

Technology became complement professional development in terms of the alternative ways of

delivering the training and the interactive materials which are provided for employees. Technology-based

training constructs “student learning community” which is a notion implies the new way of gaining learning

experiences, providing new way of communication, and facilitating the interaction between students/ instructors

ONLINE LEARNING ASSESSMENT 7

(Kochtanek & Hein, 2000). “A learning community is a group of people (students and instructors) bound

together by shared learning experiences” (Kochtanek & Hein, 2000, 282). The advocacy of learning

community is to connect the ideas in classrooms with the real world. Thus, the evaluation of professional

development program is the systematic process to assess the worth, meaning and value of a programme (Ghosh

et al., 2012). Finally, gaining learning experiences through integrated technology into professional

development can be delivered by different approaches through online learning. For example, it can be delivered

through self-paced learning and collaborative learning. Those approaches require that students be capable to

regulate their own learning.

Online Learning. The widespread of employing educational technology into educational situations a

long with pedagogical learning models enhance delivering learning through different approaches. Dabbagh

defined online learning as “an open and distributed learning environment that uses pedagogical tools, enabled

by The Internet and Web-based technologies, to facilitate learning and knowledge building through meaningful

action and interaction.” (2005, p. 164). In addition, online learning is defined as a process of learning

capitalizing the instructional materials and activities which are delivered through the internet and online

technologies, and it occurs by teachers’ facilitation in helping learners to achieve their intended objectives.

Moreover, several studies (Cook & Steinert, 2013; & Milligan & Buckenmeyer, 2008) summarized the

characteristics of online learning that learners must master, into four points: (1) motivated, active and

autonomous learning, (2) supports independent learning, (c) bolsters self- regulation, and (d) increases verbal

and written communication capabilities. As a result, the learner is a self-directed learner on the online learning

environment.

Collaborative Learning. “Collaboration is a process of mutual engagement and interaction to create

shared knowledge which may not be achieved otherwise, and is the collective effort of the members that may

result in something more than the sum of its parts” (Kwan & Yunus, 2014, p.60). In addition, collaboration

ONLINE LEARNING ASSESSMENT 8

learning capitalizes the affordances of widespread social media such as discussion forums and Wiki. For

instance, Kwan and Yunus (2014) explained that Wiki supports collaborative learning, where it is a group

working and the participation can be monitored. Finally, collaborative learning encourages the group of

students to set the goals of their tasks and to monitor and evaluate their learning within the group.

Self-Regulation (SR). It is defined as a cyclic process through which students self-perform a task, plan

it, carry out, monitor, and assess their learning performance and identify the causes of the outcomes (Panadero,

Tapia, & Huertas, 2012). The researchers recognized the importance of SR by focusing on the roots of

individual differences such as self-efficacy, motivational goals, and self-assessing the performance of their own

learning process. Generally, self-regulated learning (SRL) refers to an individual’s ability to understand and

control his or her own learning environment (Dabbagh & Bannan-Ritland, 2005).

Zimmerman defined SRL is the degree in which learners are active participators in their learning process

(2008). The abilities in self-regulation learning involves goal settings, self-mentoring, self- instruction, and

self- reinforcement. Therefore, self-regulated learning (SRL) deemed self-directive processes and skills, the

learner transforms the mental ability to skills (Zimmerman, 2008). Thus, the self-regulated learning does make

less emphasize on learner’s mental ability. The formal theories of social cognition, metacognition, and

behaviorism emphasizes that the three properties (metacognitive ability, motive, behavior) are core elements for

self-regulated learning (Zimmerman, 2008). Therefore, Self- regulated learning strategies focus on three

components of content domain which are: cognition, metacognition, and motivation. The formal theory of

metacognition consists of two components: knowledge of cognition and regulation of cognition (Schwartz,

2010). It relies on three sources which are cultural learning, individual construction, and peer interaction. The

regulation of cognition refers to the activities that support learners to control their own thinking and learning. It

includes three essential skills: planning, monitoring, and evaluation. Evaluation refers to examining the

performance, products, and regulated learning processes.

ONLINE LEARNING ASSESSMENT 9

At the end, all the definitions of SRL assert the goal directed and use one or more of SRL properties, so

learners are acknowledgeable about their learning performance (Zimmerman, 2008). As a result, the construct

of SRL refers to the degree to which learners can regulate and control the aspects of belief, behavior, and

motivation through learning processes (Nicol & Macfarlane-Dick, 2006). Also, the researchers clarified that

learners in SRL require to practice different learning processes such as setting of learning goals, orientation

toward strategies that are used to accomplish goals, and management of the resources.

All of the above, self- regulation is skills which are necessary for distance learners to master them in

order to achieve the intended outcomes from learning processes.

Students Learning Community

Collaborative Learning

Self-Paced Learning

Online Learning

Self-Regulation

Self-Regulation in Students’ learning Communities

ONLINE LEARNING ASSESSMENT 10

The Implication of Online Learning Assessment and Self- Regulation

In recent years, integrating technology into teaching and learning has become a convenient pedagogical

strategy for human learning and development. Dabbagh and Kitsantas (2012) summarized how social media

supported self-regulated learning by illustrating the pedagogical framework of social media by Zimmerman's

model. For instance, students are requested to use Wiki for organizing and managing content, encouraging

collaborative editing and commenting on peer participations, and promoting self-evaluation of their learning

across time. Thus, assessing a student’s learning achievement is a fundamental aspect of instruction. However,

this assessment deemed a challenge occurs with the online learning. For instance, Kearns (2012) realized that

the significant area of focus for assessing students’ learning through online courses is the instructor’s method of

formative and summative assessment; specifically, the distribution of grades across activities for an entire

course and the provision of effective feedback. Therefore, Kearns (2012), conducted a study in two phases to

explore and examine the practices and challenges among a group of instructors delivering online courses in a

university. The construction of the online course assessment involved, in the first phase, a written assignment,

an online discussion, a fieldwork, tests, quizzes, exams, and a presentation. The second phase involved an

online focus-group session and telephone interviews with instructors of the online courses. The findings of the

study emerged with three themes: (a) the physical separation between students and instructors, (b) time

management and workload, and (c) the challenges in assessing students in communication through technology.

Kearns (2012), mentioned that a rubric becomes the effective method for assessing students’ learning in large

assignments, grading a target performance, and facilitating instructors in grading. The rubric is defined as “a

scoring guide that lists criteria against which assignment submissions will be evaluated” (p.203). Similarly,

another study was carried out to explore the method of assessing students in the process and the product of

online collaborative courses (MacDonald, 2003). The study was applied to two online courses in UK

universities which varied in the degree of integrating online collaboration in the courses. Online collaboration

ONLINE LEARNING ASSESSMENT 11

was the central form of the course which constructed the first two blocks and linked to the assessment, where

students were divided into 4-6 groups to interact with each other under the tutorial supervision, and they

engaged in online discussion and used conferences for the rest of the course. To assess the process of online

collaborative learning, students were required to submit five transcripts of their own contributions to an online

debate to demonstrate their participation individually and on peers commenting as well. The finding of

assessing collaborative learning showed that the assignment of online collaborative learning had a positive

impact on motivating students in engaging in collaboration, improved quality of contribution, and developed

online group working. Moreover, to assess the product of online collaborative learning, students were required

to submit a report of a critique of a fictional newspaper article. The other finding of the study with respect to the

product was “attracting individual marks” (MacDonald, 2003, 286). The significant aspect of the online

collaborative learning assessment was the transcript, which was deemed a useful assessment tool in several

ways. However, online collaborative assessment might lead to a lack of flexibility in the dependency on

groups. Overall, assessing learners through online learning environment is a challenge and it needs for

instruments in order to support instructors in assessing learners’ performance.

The Development of Goal Setting and Self- Assessment in Self- Regulated Learning

As suggested earlier, SRL promotes learners to practice the processes of regulation into different aspects

and support them to reflect on that practice and experiences of learning. Nicol and Macfarlane-Dick (2006)

identified that learners in SRL engage in monitoring gaps between the internal task goals and the external

outcomes that they produced. Therefore, the researcher suggested that teachers are required to build structured

opportunities for self-monitoring and assessing the progress of learning goals. The recommended activity to

promote reflection on learning progress and experience is self-assessment. In the last decades, self-assessment

has been the significant interest in higher education. It is a crucial process in self- regulation, where learners

compare their own performance and process with a certain criteria in order to understand what they achieved

ONLINE LEARNING ASSESSMENT 12

and what they need for change and to learn from the mistakes to perform better in future tasks (Panadero et al.,

2012). The criteria of self-assessment must be visible for students from the beginning of the learning process to

promote students to define the learning goals and plan accordingly and perform the intended tasks. There are

two types of self-assessment criteria: a rubric and a script. The rubric has three essential characteristics: a list of

criteria and standards to assess the goals of the tasks, a scale of grading, and a description of each qualitative

level. Thus, students can compare and grade their performance according to the rubric. Scripts “are specific

questions structured in steps to follow the expert model of approaching a task from beginning to end” (Panadero

et al., 2012, p. 807). This kind of self-assessment tasks is dependent on the goals of the task and teachers’

instructions and supports SRL.

In addition, self-regulation refers to the Self-Control Theory, highlights three core components of self-

regulation: goal setting, goal monitoring, and goal operation (Burnette, O’Boyle, VanEpps, Pollack, & Finkel,

2012). All these three processes are conceptualized and manifested into implicit theoretical literatures including

goal setting (performance goals and learning goals), goal operating (helpless-oriented strategies and mastery-

oriented strategies), and goal monitoring (negative emotions and expectations). The study examined the process

of goal setting with regard to the SOAM model to make a distinction between performance goals and learning

goals (Burnette et al., 2012). The learners who pursued the performance goal have demonstrated the ability

which is frequently relative to their peers. Whereas, learners who pursued the learning goals have demonstrated

mastering a skill, and usually it is an internal reflection of satisfaction.

Online Learning Assessment

First of all, summative evaluation is the impact of online materials after they are completed (Dabbagh &

Bannan-Ritland, 2005). Dabbagh and Bannan-Ritland (2005) indicated that determining the suitable methods of

evaluating technology-based training is a challenge that practitioners are encountered. They concluded that

evaluating online learning with traditional evaluation methods does not support the nature of online and

ONLINE LEARNING ASSESSMENT 13

distance education. As online learning is a constructivist learning, the evaluation of online learning is

developed under the constructivist evaluation. For instance, Dabbagh and Bannan-Ritland (2005) stated the

essential principles of the evaluation methods for online learning, which is established by the Pennsylvania

State University. They are: assessment instruments and activities must align with learning goals of the

delivered training through online learning environment; assessment and instrument strategies “ should be

integral parts of the learning experience—enabling learners to assess their progress, to identify areas for review,

and to reestablish immediate learning or lesson goals” (p.255); and the assessment instruments should be

aligned to the characteristics of distance learners, and the online learner should be given opportunities for

providing feedback on their learning progress.

Context. The project will be conducted in my country, Sultanate of Oman., specifically in the

Specialized Center for Professional Training of Teachers (SCPTT). The SCPTT is located in the capital city,

Muscat, and it is covered under the egis of the Ministry of Education. It is the overarching professional

development organization for the education workforce in Oman, which aims to:

• Develop the skills of teachers in order to improve students’ learning achievement and progress.

• Improve the quality of leadership at all levels so that the quality of learning will improve in all

schools.

• Provide policy-makers with information on the development of the education profession, and to

provide the MOE with data to support the management of the school system.

Overall, the SCPTT aims at accomplishing the above goals by providing an ongoing professional

development program for 8000 participants. For each participant, the program will last two years, with an

accredited Higher Education Diploma. The targeted groups in the SCPTT center are: subject expert trainers,

center associates trainers, supervisor trainers, and new teacher program trainers.

ONLINE LEARNING ASSESSMENT 14

Participants. The participants of this paper is the master teachers, who are trained in the (SCPTT) by

the center associate trainers to address areas for development in international comparative tests and basic

literacy. Master teachers including 1040 Mathematics teachers of grades 7-12 representing all cycle 2 and post-

basic schools, 520 Science teachers of grades 7-12 representing 520 cycle 2 and post basic-schools, and 1040

Arabic teachers cycle 1 representing all cycle 1 schools. Participants will receive 20 days of face-to-face

training, 140 hours of online learning, and 75 hours of workplace learning. The selected participants represent

over 20% of all teachers in their respective specializations.

Assessment Instruments

Several instruments have been developed for assessing SRL on constructs of cognition, metacognition,

and motivation. They including Learning and Study Strategies Inventory, Motivated Strategies for Learning

Questionnaires, Self-Regulated Learning Interview Scale, Metacognitions Questionnaire, Productivity

Measurement and Enhancement System, and selection, optimization, compensation (SOC).

Learning and Study Strategies Inventory (LASSI). First of all, Zimmerman (2008) demonstrated a

number of instruments measuring SRL under those constructs: LASSI, MLSQ, and SRLIS. The Learning and

Study Strategies Inventory (LASSI) consists of 80 items; self-report inventory is divided up into three domains.

It involves 10 scales to assess several skills of SRL including metacognitive (concentration, selecting main

ideas, and information processing), motivational (attitude and anxiety), and behavioral skills (time management,

study aids, self-test, and test strategies). The subscale using 5 points rating to rate the students’ responses from

not at all typical of me to very typical to me.

Motivated Strategies for Learning Questionnaires (MSLQ). The second instrument is a questionnaire

focused in motivated strategies of SRL (Motivated Strategies for Learning Questionnaires, MSLQ; Pintrich,

Smith, Carcia, & McKeachie, 1993). It is an 81-item questionnaire and is divided into two sections: learning

ONLINE LEARNING ASSESSMENT 15

strategies and motivation. Learning strategies is further divided into sections relevant to a cognitive and

metacognitive skills. The cognitive skills focused on, for instance, elaborate, rehearse, organized, and critical

thinking. The metacognitive skills include time management, environmental study, and effort management,

seeking for help, and learning with peer. The scale of this measurement was using 7-point rating from “not all

true of me” to “very true of me.”

Self-Regulated Learning Interview Scale (SRLIS). An additional instrument is Self-Regulated

Learning Interview Scale (SRLIS) to assess the cognitive, metacognitive, and behavior of SR. The study in

assessing SRL through online learning shows the interrelation of various processes in real time in terms of the

effect of goal setting on self-monitoring. From the open-ended questions, 14 codes, related to three components

of SL, have emerged. For example, students’ motivation shows self-evaluation reactions and self-

consequences. The metacognitive skills are demonstrated as goal setting and planning and environment

structure, which help seeking and monitoring that emerged to the behavioral aspect. SRLIS uses a 4-point scale

that ranges between seldom and all of the time. The questionnaires that are used to measure SRL through the

online event offer answers to how do students become masters of their own learning processes.

SR into a collaborative context. Furthermore, there was a recent study was conducted by Law, Ge,

and Eseryel (2016) to identify the dimensions of the construct of self-regulation in a collaborative context. The

construct examined the SR into a collaborative context in terms of three factors: (1) sets goals, (2) monitors, and

(3) reflects on his/her cognition through interpersonal interactions. One hundred and thirty-one undergraduate

students, who enrolled in the educational psychology course at a Midwestern university, administered the

assessment. The instrument that measured the self-regulation was developed based on “three theoretical

frameworks: self-regulation (Pintrich 2000; Zimmerman 2001), collaborative learning (Goos et al. 2002; Iiskala

et al.2011; Teasley & Roschelle 1993), and ill-structured problem solving (Jonassen 1997;Sinnott 1989; Voss &

Post 1988)” ( Law, Ge, & Eseryel, 2016, p. 6) . The items of the construct were adopted from existing self-

ONLINE LEARNING ASSESSMENT 16

regulation and problem-solving scales which are Motivated Strategies for Learning Questionnaire (Pintrich et

al. 1991) and a self-report questionnaire of ill-structured problem-solving skills (Ge, 2001). The points of the

scale was seven-point Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree). Correlations among

the factors were calculated to examine the relationships among various factors. The result of the assessment

showed that students clarified and elaborated their own thoughts and their understanding of other members’

thoughts based on the behaviors are shown in the items. Moreover, the result identified that students co-

regulate in a problem-solving environment by refuting others and summarizing others’ inputs. This finding

indicated that problem solvers engaged in reflection and evaluation during the problem-solving processes.

Finally, the planning component of self-regulation was aggregated in the clarification and resolution factors.

That was because the planning related constructs were not found in the final structure.

All the above three instruments are used to assess SR in three different domains, and they are defined as

the aptitude measures for SR (Zimmerman, 2008). Zimmerman (2008) described that LASSI and MLSQ are

reflective reports, whereas SRLIS conveys multiple perspectives for hypothetical learning context.

Metacognitions questionnaire (MCQ). A common measure is used to assess the metacognition of

SRL. The metacognitions questionnaire (MCQ; Wells & Cartwright-Hatton, 2004). The measurement was

originally created by the joint effort of the University of Manchester and the Norwegian University of Science

and Technology in order to assess several dimensions of metacognition beliefs. MCQ consists of a 65-item

scale and encompasses five correlated factors that assess three domains of positive and negative metacognitive

beliefs, metacognitive monitoring, and judgments of cognitive confidence. The reliability and validity of the

MCQ-30 factors are evaluated by assessing internal consistency of the items and item-total correlations of each

factor, test–retest correlations of factors, and the correlations of factors with other scales that have been

demonstrated to correlate meaningfully with the full-scale MCQ in previous studies.

ONLINE LEARNING ASSESSMENT 17

Productivity Measurement and Enhancement System (ProMES). Another measure that is used for

goals setting is the Productivity Measurement and Enhancement System (ProMES; Pritchard, Young, Koenig,

Schmerling, & Dixon, 2012). This measure is created by the University of Central Florida. The component of

the measure is the measurement/feedback system that tracks students’ accomplishment. This measurement

consists of 4–6 objectives and 8–12 indicators. ProMES integrates learning goals in at least two ways: (1) it

cultivates the productivity and improvements for long-term processes and (2) it allows learners to focus on the

overall goals. It is a design team that consists of five to eight people, and the organizational unit consists of 5–

35 members. It uses 5-point Likert-type scales. The researchers ask the participants to respond to the

questionnaires, and then meta-analysis is collected. The approach of this measurement has both advantages and

disadvantages. The advantage is that the researchers are acknowledgeable about the study; hence, they could

provide rich information more than that the rater can provide. The disadvantage of the scale is that the

participants on the study cannot be trained and cannot be evaluated to confirm the reliability of the scale. The

major challenge of this study is the large number of variables assessed by the questionnaires and there are no

complete data to cover the questionnaires variables. As the multiple R for the WLS analysis is .77, the result of

the study is that the productivity improvements are high with the ProMES intervention.

Selection, optimization, compensation (SOC). Another measurement related to self-regulation and

goal setting is SOC, which is based on three fundamental process (selection, optimization, compensation, SOC;

Pritchard et al., 2012). SOC consists of 48 items on self- report questionnaires. Validity and reliability of the

construct are examined by subsequent studies. Validity is based on the factor structure, retest reliability, and

criterion-related validity (Pritchard et al., 2012).

In Conclusion, The above instruments have both negative and positive values. The ProMES instrument

is a comprehensive instrument, but it lacks for the reliability. SOC instrument has been examined from different

researchers and it showed a positive value for measuring SR. The MCQ instrument has been examined along

ONLINE LEARNING ASSESSMENT 18

spread several studies which showed the positive values from using it in measuring SR. The instruments,

LASSI, MLSQ and SRLIS, have been used from different studies and are comprehend in measuring different

factors of SR such as planning, monitoring, self-evaluation, and reflection. At the end, measuring SR through

learning communities is in demand in order to provide evidence how learners can control, direct and regulate

their own learning into learning processes to achieve the success.

Finally, by searching for an application of online assessment and goal setting assessment in database

(Mental Measurements Yearbook), there were 25 items according to PsychTests AIM, Inc. (2011). The items

are related to several areas; however, there are no items relevant to the area of interest, which is an online

assessment in terms of assessing self –regulation, goal setting and self-assessment. The items belong to

different skills such as management skills, leadership and sport. For instance, The Management Skills and

Styles Assessment (MANSSA) consists of 196 multiple-choice items. It covers eight skills in management, and

the scores on the MANSSA scales can be used for the purposes of identifying promising managers. The test is

administered online, and the response to each item is based on Likert scales (agree to disagree) and a progress

bar to report the result of test completion. The period of the test is 40 min. It has been reviewed by three

authors: Oswald, Reilly, and Richard . Consequently, the test that is developed during this course is required to

make a better understanding of teachers’ self-regulation learning skills in terms of their ability on setting their

learning goals and assessing themselves.

At the end, the SCPTT revealed the need for further development of instruments to assess trainees’

learning performance. This conclusion was revealed from the study was conducted in the SCPTT and reported

the lack of instruments for assessing learners through online learning. In addition, several studies showed that

assessing learning through online learning is still an issue. Finally, developing instruments to assess learning

performance using a digital medium is the focus of this paper.

Proposed Solution

ONLINE LEARNING ASSESSMENT 19

Assessing teachers in a collaborative learning through the SCPTT encountered challenges, this paper

will focus on measuring the metacognitive skills of self-regulation learning in terms of (a) goal setting, and (b)

teachers’ self-assessed abilities in order to ensuring the quality of learning; developing standards of assessment

for online courses; and attaining the validity of the assessment. The group that administered this test were the

master teachers who are trained in advanced teaching methodologies, professional skills, higher –order-

thinking skills and 21 the century skills. The overall goals are to ensure the quality of learning; developing

standards of assessment for online courses; and attaining the validity of the assessment. The number of

participants for the project was 28 participants. The pilot test was developed as a summative assessment.

Assessment Development

Constructs of the measurement

Nowadays, there are emergent models of teachers’ professional development that engaging teachers in

collaborative learning to making a meaningful shifting of teaching practices into classrooms. Collaborative

learning provides learning communities where teachers can discuss the theories’ principles and pedagogical

models, share instructional strategies and experiences into classrooms, and participate in discussion forums

collaboratively with different teachers to exchange ideas and feedback about their application into classrooms.

All those processes collectively put teachers into authentic learning that lead them to discover the learning from

multiple resources rather than only from one resource which is the instructor. Therefore, collaborative learning

of professional development of teachers is more about supporting teachers to construct a conceptual knowledge

of teaching and improving their practices in classrooms. Thereby, students’ learning performance will be

improved.

The evaluation process in the (SCPTT) is based on different aspects. They are (a) evaluating trainees,

(b) evaluating training programs, and (c) accreditation. Evaluating trainees is based on the retain of investment

ONLINE LEARNING ASSESSMENT 20

(ROI) in five different levels which are: assessing the satisfaction in face-to-face training, assessing learning

performance through online learning, assessing application in workplace, assessing students’ learning

performance, and assessing (ROI) through analysis of the impact of finance and human resources that been

invested in the project.

Conceptual Framework and Construct Measurement

Self-regulation learning emerged in the 1980s and continues to be explored by several researchers and

educators with diverse background until 21the century in order to answer the question of how students master

their own learning process at their own pace (Zimmerman, B. J., 2008). Zimmerman (2008) pointed that the

construct of self-regulation learning refers to the self-directive process which promotes learners to transfer their

mental abilities into academic skills. That leads them to be proactive learners and to have the capability to

manage and organize their own learning process rather than receiving and absorbing knowledge from the

instructor as passive learners. In addition, he explained the self-regulation learning as a social form of learning

rather than isolated learning, where learner seeks support from peers, coaches and teachers. Thereby, the self-

regulation learning emphasizes on three core elements which are: a metacognitive ability, a motivation ability,

and a behavioral ability.

Zimmerman (2000) conceptualized the SRL into three phases as a cyclic model to understand how and

why learners achieve learning. The first phase is called the forethought phase that includes task analysis (goal

setting and planning) and self-motivation beliefs (self-efficacy, task interest, goal orientation, and outcomes

expectation). The second phase is called performance phase that includes self-control (self-instruction,

imagery, and task strategies) and self-observation (self-recording and metacognitive monitoring). The third

phase is called self-reflection that emphasizes self-judgment (self-evaluation and casual attribution) and self-

reaction (satisfaction, and adaptive and defensive).

ONLINE LEARNING ASSESSMENT 21

In order to measure the students’ self-regulation in terms of goal setting and self-assessment, we will

understand how and why students achieve academically and achieve the desired goals. Also, it will explain how

students control and organize their own learning to success in their learning. In addition, measuring self-

regulation will acquire students the strategies and tools to assess their own performance and to set their learning

and performance goals. The selection of those components of SR, goals setting and self-assessment, are

essential skills for learner who learn through technology medium weather through the online learning,

independent learning, and collaborative learning. Moreover, studying trainers experiences in SCPTT showed

that instructors lack for instruments and criteria to assess their trainees’s learning progress through online

learning.

Description of the test

The test type that will be used to assess the self- regulated learning, goal setting and self- assessment, is

a summative assessment. It was conducted at the conclusion of a unit of an online learning course to summarize

the strengths and weakness of the instructions and teachers’ performance. The assumption from developing the

summative assessment of this paper is to provide information about teachers’ mastery of the training materials

after instruction and practice of the unit are completed at a given point in time. It included multiple choice

questions, and a performance task. The unit is taught within 2 months (8 weeks) through an online learning

mode. The summative test was conducted electronically through the web- based assessment via Survey

Monkey. Teachers were asked to select one response from three to four choices in the MCQ. In addition,

teachers were evaluated on the quality of contributions on the discussion forum by illustrating on the criteria of

the rubric, and they were judged on the performance task by writing a report which will focus on the quality of

their final performance based on constructed- responses. The following chart identifies the structure of the

assessment.

ONLINE LEARNING ASSESSMENT 22

Purpose for the test

The purpose of the test is to evaluate teachers’ ability in goal setting and self –assessment skills. Goal

setting skills for teachers in the (SCPTT) are assessed based on their abilities if they have set their goals in order

to perform the learning tasks? What are the strategies and practices they often do to approach their tasks through

Assessment

SR: to measure goal setting and self- assessment

Classification Schemes

Performance Task

50 Points

Numerical Scales (MCH)

50 points

Tests

Masters Teachers who are taking professional development online

Evaluation

Decision Making

ONLINE LEARNING ASSESSMENT 23

online learning? What are the criteria they rely on in order to participate in an online discussion? However, in

the performance task, the teachers are assessed if they have the skills of self-assessment in terms of do they

have reached their goals from the training programs? What do they want to improve? And where they see

themselves in future in developing their teaching skills? In general, the summative test provided insights about

the status of the students’ learning performance after teachers are completed the period of the training program.

Finally, this assessment is a critical assessment to evaluate the effectiveness of the professional training of

teachers that are designed in the (SCPTT) to determine its’ weakness and strengthens for future modification

and improvement.

Inferences and Decisions

The data of the test will be presented in tables. The information and data that will be obtained from the

test scores will show indicators about the effectiveness of the professional training programs which deliver

through the (SCPTT). That is because the content of the test’s items focused on two components of SR, goal

setting and self-assessment. Therefore, the finding of the data analysis of the test will add value to the

performance in the SCPTT. Specifically, results of the test will provide data about teachers’ performance in

terms of do they manage and organize their learning process within the determined period, what is their

feedback about the skills they learned from the training materials, are they motivated to improve their

profession in the future, how they applied knowledge and skills that they gained from the professional training

in classroom, do teachers see an improvement in students’ learning performance and their motivation by

applying different teaching methodologies, what are the kinds of the activities should be modified in the next

training materials.

According to the data which are obtained from the test scores, the (SCPTT) will make a decision to

improve the training materials of the training program, to design a variety of learning tasks that required more

ONLINE LEARNING ASSESSMENT 24

application into a classroom as well as authentic learning activities to allow teachers transfer the learning

outcomes into real- life settings. In addition, the (SCPTT) will consider adding other skills in the training

materials for self-regulation strategies to ensure that teachers are capable of continuing to develop their teaching

skills and setting their own goals after completing the training program.

The type of decision that can be made based on the obtained data is:

(a) Data Driven Decision Making: the data of the test will be used to improve the instructions relevant to the

training materials and to improve the teachers’ performance.

Report of Data

The total score of each participant was obtained to demonstrate the general level of self-regulation. The

scores for the two factors, goal setting and self-assessment, were reported independently to show students’

acquisition of features for the SR. Then, the scores were compared in order to define the level of SR among the

participants.

The data analysis of items difficulty and items discrimination were used in order to examine possible

relationship between scores of the participants in SR. Also, the reliability, Central Tendency and variability are

analyzed.

Running head: ONLINE LEARNING ASSESSMENT 25

Plan of the Test of SR (Goal Settings and Self –Assessment).

Content Type of Item Numbers of

Items

Points per item

(Weighting)

Total points

Self- regulation MCQ 10 5 50

Self-

Assessment

5 5 25

Goal Settings 5 5 25

A report PBT 1 50 50

Total 2 types 21 100

Test

Multiple choice questions of self- regulated learning (goal setting):

Read the following questions, click only the correct answer.

1. Goal setting focuses on how goals are related to job performance, which of the following represents the

practices of goal setting for your learning task:

A. Ask peers to share the timeline.

B. List all resources.

C. Take your time to make a better understanding of the task first.

ONLINE LEARNING ASSESSMENT 26

2.  Once you are participating in a discussion forum to answer ill-structured problem, what skills should

you use to make accomplishment on an online learning within a period of two months?

A. Study extra materials.

B. Set goals for yourself.

C. Manage your time.

D. Study with partners

3. When you posting your response in the discussion forum, what criteria are you looking for:

A. Accuracy of your responses.

B. The aesthetic of the responses.

C. The number of responses for peer.

D. The number of words of others.

4. What are the strategies are you using to start your learning activities in the discussion forum:

A. Develop your learning plan.

B. Manage your time.

C. Study all the time.

D. Wait for the instructors’ notification.

5. What strategies are you using to evaluate your performance in the discussion forum?

A. Ask the instructor.

B. Check the criteria of the rubric.

C. Compare my participation to peer’s participation.

D. I asked questions when I don’t understand

ONLINE LEARNING ASSESSMENT 27

6. The instructor posted an assignment in the discussion forum. In order to respond to the assignment, you

have to complete all of the reading materials during 4 days only for this task. How are you going to set

you learning goal to perform the task of the assignment?

A. I always just started perform the task.

B. I can usually plan how to reach the learning goal by writing down the action plan.

C. I developed an action plan and I revised it sometimes.

D. Often, I developed an action plan on my mind so, I work on the task when I remember.

7. In addition to use a rubric to assess your performance on an online discussion, what else you will do:

A. Compare myself to peer.

B. Perform the task directly.

C. Using a rubric only.

8. Collaborative assignments through a teamwork have begun to rise organizational customers’

satisfaction. The managers have modified how they view and practice goal setting. There are unique

complications that makes goal setting in a teamwork environment more complex. In order to facilitate

team’s accomplishment toward the goals, what are the strategies the managers can provide (Redmond,

2016):

A. Ask the manager.

B. Utilize electronic dashboard.

C. Work individually.

9. When you developed the timeline of your learning task, what is the best approach to start your timeline:

A. Finish reading all materials.

B. Start developing open plan.

C. Start with checklist.

ONLINE LEARNING ASSESSMENT 28

10. What strategies often motivate you to learn?

A. The long period will be taken to study the materials.

B. The students’ motivation in the class.

C. The interactivity of the training materials.

D. The methods of learning.

Answer Key:

1. (B) List all resources.

2. (B) Set goal for yourself.

3. (A) Accuracy of your responses

4. (A) Develop your learning’s plan.

5. (B) Check the criteria of the rubric.

6. (B) I can usually plan how to reach the learning goal by writing down the action plan.

7. (C) Using the rubric only.

8. (B) Utilize electronic dashboard.

9. (C) Start with checklist.

10. (C) The interactivity of the training materials.

Performance based task:

1. As you have attended conferences, online courses, or online tutorials, Write an essay presenting the

following questions:

Have you reached your goals?

Do the skills and strategies you learned from the training materials worked with you?

What didn’t work?

ONLINE LEARNING ASSESSMENT 29

What do you want to improve?

Where do you see yourself in the future in developing your teaching skills?

Rubric

A Rubric for the report evaluation

Criteria  Excellent Good Average Poor

Areas of

Improvement

Very clear strengths and

weakness, very clear

recommendations

Clear strengths and

weakness, clear

recommendations

Vague information,

some

recommendations

Absent,

Very little

recommendation

Quality of

responses

very clear responses,

illustrate examples

clear responses,

examples exist,

clear responses, no

examples.

not clear

responses, no

examples

Length of

words

3 paragraph 2 paragraph 1 paragraph  few sentences 

Points 50-40 30-20 20-10 10 or less

Pilot Testing

The present assessment paper is a class assignment. It aims to develop and administer a pilot test of

online learning assessment. The purposes of the assignment are to understand the assessment processes, to

analyze data from a pilot test, and make adjustments to the test. An Institutional Review Board (IRB) approval

is not required for this study, since it is not going to be published. However, an IRB approval is valuable for

students in order to practice the process. Therefore, I drafted a letter to get permission from the IRB. The

following is the draft of the letter:

ONLINE LEARNING ASSESSMENT 30

“I would like to develop an assessment about the metacognitive components of self-regulated learning in

terms of (a) goal setting and (b) teachers’ self-assessed abilities. The group that will take up the test consists of

master teachers, who are trained in advanced teaching methodologies, professional skills, higher thinking skills,

and 21st-century skills. The overall objectives of the test are (1) to ensure the quality of learning; (2) develop

standards of assessment for online courses; (3) and to attain the validity of the assessment. The number of

participants is 28. My interest in developing the pilot test is based on my specialization in Instructional Design

and Technology, which includes different pedagogical models of learning. Self-regulated learning is an

important factor that controls individuals’ learning at their own pace, but it is a challenge to assess these

learners who take up online courses in terms of whether they have the skills and abilities to manage their own

learning and also to manage their goal setting in order to achieve the intended skills. Thus, I am interesting in

understanding the assessment processes in this area. I will implement the pilot test at the Specialized Center for

Professional Training of Teachers (SCPTT) in my country, Sultanate of Omani. The purpose of focusing on the

proposed group is that these master teachers who take up training through online courses have earlier mentioned

in their feedback on a qualitative study, which was part of my doctor class assignment, that there was a lack of a

clear description of the evaluation criteria. Therefore, I am interested in evaluating their performance and

learning tasks through the online courses. Hence, developing the pilot test will help in gaining knowledge on the

process of developing an assessment for online courses and thereby would enhance my future career. In

addition, it is beneficial to improve the assessment processes at SCPTT, as it is a new project. The data of the

pilot test is only a class assignment. I also declare that I have read the documents of classroom projects, and the

main purpose for the request of this approval to this particular assignment is to administer the test in my

country, Oman, and through online medium”.

Data Collection

ONLINE LEARNING ASSESSMENT 31

The data of the study was collected online through technology medium, i.e., an online survey software.

SurveyMonkey was used as a medium for data collection. Ten multiple choice questions (MCQ) and a

performance task were developed. The content of the multiple choice questions focused on the skills and

abilities for goal setting, whereas the performance task aimed to assess participants’ abilities in assessing

themselves. The responses of the performance task were graded by using the rubric.

A link for the survey (test) was sent to participants through email and WhatsApp. The participants

responded to the survey over the course of 2 weeks. I have not received clarification from the participants

regarding the questions of the test. Table (1)

shows an example of participants’ responses

to MCQ.

Table (1)

Data Analysis and Findings

For the present measurement, the test was carried out through SurveyMonkey, and the summary data

obtained as different versions like chart, graph, or numerical table. Spreadsheet and SPSS software are used to

analyze the data of the test as frequency, scale, reliability, items difficulty and discrimination, and coefficient

correlation. Chart 1 shows an example of the analysis that was obtained from the SurveyMonkey.

Goal Setting

What strategies  are you using to start your learning activities in the discussion forum:

Answer Options Response Percent Response Count

A. Develop your learning plan. 70.4% 19B. Manage your time. 18.5% 5C. Study all the time. 3.7% 1D. Wait for the instructors’ notification. 7.4% 2

answered question 27skipped question 1

ONLINE LEARNING ASSESSMENT 32

Chart (1): Individual Responses on multiple choice questions and constructed response.

ONLINE LEARNING ASSESSMENT 33

ONLINE LEARNING ASSESSMENT 34

ONLINE LEARNING ASSESSMENT 35

Chart (2) Responses of performance task.

The educational and psychological researchers contended that measurements must be conducted on a

number of variables (Johnson & Christensen, 2014). According to the summary data of the above SR

measurement, there were participants who have skipped the questions. Three participants have skipped the first,

ONLINE LEARNING ASSESSMENT 36

second, and the third questions of MCQ, and 16 participants have skipped the performance task, where only

twelve participants have reflected on it. The initial results show that these participants didn’t have the skills of

goal setting and of self-regulated learning, or else it would also be that they might have had unsuccessful online

learning experiences.

Statistics

Distributions. The following table shows raw scores and frequencies of participants’ responses.

GOAL SETTING

Frequency Percent

Valid

Percent

Cumulative

Percent

Valid 1 1 3.4 3.6 3.6

2 1 3.4 3.6 7.1

3 1 3.4 3.6 10.7

4 1 3.4 3.6 14.3

5 1 3.4 3.6 17.9

6 1 3.4 3.6 21.4

7 1 3.4 3.6 25.0

8 1 3.4 3.6 28.6

9 1 3.4 3.6 32.1

10 1 3.4 3.6 35.7

11 1 3.4 3.6 39.3

12 1 3.4 3.6 42.9

13 1 3.4 3.6 46.4

ONLINE LEARNING ASSESSMENT 37

14 1 3.4 3.6 50.0

15 1 3.4 3.6 53.6

16 1 3.4 3.6 57.1

17 1 3.4 3.6 60.7

18 1 3.4 3.6 64.3

19 1 3.4 3.6 67.9

20 1 3.4 3.6 71.4

21 1 3.4 3.6 75.0

22 1 3.4 3.6 78.6

23 1 3.4 3.6 82.1

24 1 3.4 3.6 85.7

25 1 3.4 3.6 89.3

26 1 3.4 3.6 92.9

27 1 3.4 3.6 96.4

28 1 3.4 3.6 100.0

Total 28 96.6 100.0

Missing System 1 3.4

Total 29 100.0

practices of setting goals

Frequency Percent

Valid

Percent

Cumulative

Percent

ONLINE LEARNING ASSESSMENT 38

Valid 1 9 31.0 33.3 33.3

2 10 34.5 37.0 70.4

4 8 27.6 29.6 100.0

Total 27 93.1 100.0

Missing NO

DATA

1 3.4

System 1 3.4

Total 2 6.9

Total 29 100.0

what skills should you use to make accomplishment on an online

learning within a period of two months?

Frequency Percent

Valid

Percent

Cumulative

Percent

Valid 1 16 55.2 57.1 57.1

2 10 34.5 35.7 92.9

3 1 3.4 3.6 96.4

4 1 3.4 3.6 100.0

Total 28 96.6 100.0

Missing System 1 3.4

Total 29 100.0

what criteria are you looking for:

ONLINE LEARNING ASSESSMENT 39

Frequency Percent

Valid

Percent

Cumulative

Percent

Valid 1 18 62.1 66.7 66.7

2 5 17.2 18.5 85.2

3 1 3.4 3.7 88.9

4 3 10.3 11.1 100.0

Total 27 93.1 100.0

Missing NO

DATA

1 3.4

System 1 3.4

Total 2 6.9

Total 29 100.0

strategies are you using to start your learning activities in the

discussion forum

Frequency Percent

Valid

Percent

Cumulative

Percent

Valid 1 3 10.3 10.7 10.7

2 8 27.6 28.6 39.3

3 8 27.6 28.6 67.9

4 9 31.0 32.1 100.0

Total 28 96.6 100.0

Missing System 1 3.4

ONLINE LEARNING ASSESSMENT 40

Total 29 100.0

strategies are you using to evaluate your performance in the

discussion forum

Frequency Percent

Valid

Percent

Cumulative

Percent

Valid 1 1 3.4 3.6 3.6

2 15 51.7 53.6 57.1

3 8 27.6 28.6 85.7

4 4 13.8 14.3 100.0

Total 28 96.6 100.0

Missing System 1 3.4

Total 29 100.0

How are you going to set you learning goals to perform the task of

the assignment

Frequency Percent

Valid

Percent

Cumulative

Percent

Valid 1 15 51.7 53.6 53.6

2 11 37.9 39.3 92.9

3 1 3.4 3.6 96.4

4 1 3.4 3.6 100.0

Total 28 96.6 100.0

ONLINE LEARNING ASSESSMENT 41

Missing System 1 3.4

Total 29 100.0

In addition to use rubric to assess your performance on an line

discussion, what else you will do

Frequency Percent

Valid

Percent

Cumulative

Percent

Valid 1 15 51.7 53.6 53.6

2 10 34.5 35.7 89.3

3 3 10.3 10.7 100.0

Total 28 96.6 100.0

Missing System 1 3.4

Total 29 100.0

In order to facilitate team’s accomplishment toward the goals,

what are the strategies managers can provide

Frequency Percent

Valid

Percent

Cumulative

Percent

Valid 1 6 20.7 21.4 21.4

2 3 10.3 10.7 32.1

3 19 65.5 67.9 100.0

ONLINE LEARNING ASSESSMENT 42

Total 28 96.6 100.0

Missing System 1 3.4

Total 29 100.0

When you developed the timeline of your learning task, what is

the best approach to start your timeline

Frequency Percent

Valid

Percent

Cumulative

Percent

Valid 1 6 20.7 21.4 21.4

2 5 17.2 17.9 39.3

3 17 58.6 60.7 100.0

Total 28 96.6 100.0

Missing System 1 3.4

Total 29 100.0

When you developed the timeline of your learning task, what is

the best approach to start your timeline

Frequency Percent

Valid

Percent

Cumulative

Percent

Valid 1 1 3.4 3.6 3.6

2 3 10.3 10.7 14.3

3 18 62.1 64.3 78.6

4 6 20.7 21.4 100.0

ONLINE LEARNING ASSESSMENT 43

Total 28 96.6 100.0

Missing System 1 3.4

Total 29 100.0

Measure of Central Tendency and Variability. The measure of Central Tendency was based on the

average of a set of scores with a distribution of scores. It is an average of mean, mode, and median. The

measure of variability refers to how much variation occurs among the data values. The common indexes of

variability that show the results of data analysis are range, variance, and standard deviation. Table 1 shows the

participants’ responses to the multiple-choice questions.

Table 1: Descriptive Statistics for Respondents in MCQ.

ONLINE LEARNING ASSESSMENT 44

Item Statistics

Mean Std.

Deviation

N

practices of setting

goals

2.19 1.201 26

what skills should you

use to make

accomplishment on an

online learning within a

period of two months?

1.42 .578 26

what criteria are you

looking for:

1.50 .906 26

strategies are you using

to start your learning

activities in the

discussion forum

2.77 1.032 26

strategies are you using

to evaluate your

performance in the

discussion forum

2.58 .809 26

ONLINE LEARNING ASSESSMENT 45

How are you going to

set you learning goals

to perform the task of

the assignment

1.54 .761 26

In addition to use

rubric to assess your

performance on an line

discussion, what else

you will do

1.50 .648 26

In order to facilitate

team’s accomplishment

toward the goals, what

are the strategies

managers can provide

2.50 .812 26

When you developed

the timeline of your

learning task, what is

the best approach to

start your timeline

2.38 .852 26

ONLINE LEARNING ASSESSMENT 46

When you developed

the timeline of your

learning task, what is

the best approach to

start your timeline

3.12 .588 26

According to the tables, the difference in the number of respondents between the tables of statistics and

item statistics refers to the fact that there were a few students who did not respond to all the questions. The data

analysis showed that the average mean among scores was between .690 and 1.228. The highest score was for

Q1 (2.19), and the lowest score was for Q2 (1.40). That indicates the respondents lack for time management

skills. However, they had the ability to set goals for their learning. There is a contradiction among responses

that could be a result of participants’ not understanding the questions. Hence, mean is the best measurement of

Central Tendency because it provides precise data and evidence. Also, it measures the magnitude of all scores.

However, the median is a better choice for showing “what is typical” on Central Tendency when the data are

highly skewed because of the need for describing the measures of Central Tendency, which are less affected by

extreme scores (Johnson & Christensen, 2014).

Regarding the measures of variability, the numerical index provided information that showed that the

numbers were homogenous. The standard deviation (SD) was positive across all scores, and there was no zero

(no identical scores). This shows the variability was high among scores. The SD is 8.226, and the mean is

14.50, which highlighted a large variation between them. Revision of the questions is necessary to examine the

variability.

Item Difficulty and Discrimination for Multiple Choice Items

ONLINE LEARNING ASSESSMENT 47

Item Difficulty. The item difficulty refers to the percentage of students who answered the items of the

test correctly. The percentage should be between 60% and 80%. The ratio of item difficulty has to be between 0

and 1.0. The optimal ratio is 0.5. The following table (3) shows the item difficulty.

Table (3)

Scoring

groups

Studen

t q1 q2

q

3 q4

q

5 q6 q7 q8

q

9 q10

Total

scoring%

Highes

t

1 70% 0 1 0 0 1 1 1 1 1 1

2 60% 1 0 1 0 1 1 0 1 0 1

3 60% 1 1 1 1 1 0 0 0 1 0

Total High 2 2 2 1 3 2 1 2 2 2

4 50% 1 1 1 0 0 0 0 0 1 1

5 50% 0 1 1 1 1 0 0 0 0 1

6 50% 1 0 1 1 1 0 0 0 0 1

7 50% 0 1 0 0 1 1 1 0 0 1

8 50% 1 0 1 0 1 0 0 0 1 1

9 50% 1 1 1 0 0 0 0 0 1 1

Middle 10 40% 0 1 1 0 1 0 0 0 1 0

11 40% 0 0 1 0 0 1 0 0 1 1

12 40% 0 0 1 0 0 0 1 0 1 1

13 40% 1 0 1 0 1 1 0 0 0 0

ONLINE LEARNING ASSESSMENT 48

14 40% 0 1 1 0 1 1 0 0 0 0

15 30% 1 0 0 0 1 0 0 0 0 1

16 30% 0 0 1 0 0 0 0 0 1 1

17 30% 0 0 0 0 1 1 0 0 1 0

18 30% 0 0 0 0 0 1 0 0 1 1

19 30% 0 1 0 0 1 0 0 0 1 0

20 30% 0 0 1 0 0 1 0 0 0 1

21 30% 1 0 0 0 1 0 0 1 0 0

22 30% 0 0 1 0 0 0 0 0 1 1

23 30% 0 0 0 0 0 1 0 0 1 1

24 30% 0 1 0 0 0 0 0 0 1 1

25 30% 0 0 1 0 0 0 0 0 1 1

Middle 6 6 10 1 10 8 2 1 13 14

Lowest

26 20% 0 0 1 0 0 0 0 0 1 0

27 20% 0 0 1 0 1 0 0 0 0 0

28 20% 1 0 0 0 0 1 0 0 0 0

Total Low 1 0 2 0 1 1 0 1 1 0

Discriminatio

n 0 .04 0 .04 0 .04 .04 0 0 .04

Total correct 9 8 14 2 14 11 3 4 15 16

Difficult .32 .3 .5 .1 .5 .4 .11

.1

4 .5 .6

ONLINE LEARNING ASSESSMENT 49

The correct responses to questions 3, 5, and 9 are optimal in the range of 0.5. This shows that those

items were easy for participants. The rest of the responses were in the range of (0.11) and 0(.4), which mean

the ratio is less than (>.5), and the items were difficult for several reasons. It could be that the items were too

difficult or confusing, or the items were miskeyed.

Item Discrimination. Discrimination refers to how well the item serves to discriminate between

students with high skills and knowledge of goal setting or who received the high scores to students with low

skills and knowledge of goal setting or who received the low scores. Within a group, the discrimination index

refers to the differences between a group of high-scoring students who responded with the correct answers and a

group of low-scoring students who responded with the incorrect answers. Generally, 0.30 or greater shows a

good discrimination. According to Table 3, all of the ratios showed a positive discrimination index, and the

ratio was between 0 and 1. That is, for the given test questions, the participants knew the content of the test,

and they responded with the correct answers. The responses to the items were between 0 and 0.04, which

showed a good discrimination. Finally, the responses per student are in the range of 0 and .04, which indicates

that the items of the test were not too difficult, and the students knew the content of the items; therefore, they

answered correctly. The data of the table (3) does not indicate a negative discrimination index, so almost all the

students knew the content of the test.

Reliability and Validity

Reliability. To ensure the reliability of the measure, internal consistency reliability will be used for this

construct. That is because the group of individuals was required to take the test one at a time. Therefore, the

coefficient alpha measured the reliability of the test scores. The reliability coefficient of a set of test scores is

(.330), which indicates that there is a high relationship between all responses. This result indicated that the test

was poorly developed, and it needed to be dropped or replaced. Therefore, test homogeneity is recommended to

ONLINE LEARNING ASSESSMENT 50

measure the internal consistency of the items’ scores because it provides precise test score interpretation. The

following data shows the reliability and validity of the test scores.

Case Processing Summary

N %

Cases Valid 26 89.7

Excludeda 3 10.3

Total 29 100.0

Reliability Statistics

Cronbach's

Alpha

Cronbach's

Alpha Based

on

Standardized

Items N of Items

.330 .382 10

Item Statistics

Mean

Std.

Deviation N

ONLINE LEARNING ASSESSMENT 51

practices of setting

goals

2.19 1.201 26

what skills should you

use to make

accomplishment on an

online learning within a

period of two months?

1.42 .578 26

what criteria are you

looking for:

1.50 .906 26

strategies are you using

to start your learning

activities in the

discussion forum

2.77 1.032 26

strategies are you using

to evaluate your

performance in the

discussion forum

2.58 .809 26

How are you going to

set you learning goals

to perform the task of

the assignment

1.54 .761 26

ONLINE LEARNING ASSESSMENT 52

In addition to use

rubric to assess your

performance on an line

discussion, what else

you will do

1.50 .648 26

In order to facilitate

team’s accomplishment

toward the goals, what

are the strategies

managers can provide

2.50 .812 26

When you developed

the timeline of your

learning task, what is

the best approach to

start your timeline

2.38 .852 26

When you developed

the timeline of your

learning task, what is

the best approach to

start your timeline

3.12 .588 26

ONLINE LEARNING ASSESSMENT 53

Item-Total Statistics

Scale Mean if

Item Deleted

Scale

Variance if

Item Deleted

Corrected

Item-Total

Correlation

Squared

Multiple

Correlation

Cronbach's

Alpha if Item

Deleted

practices of setting

goals

19.31 8.862 -.040 .395 .414

ONLINE LEARNING ASSESSMENT 54

what skills should you

use to make

accomplishment on an

online learning within a

period of two months?

20.08 9.194 .141 .218 .304

what criteria are you

looking for:

20.00 8.800 .074 .467 .329

strategies are you using

to start your learning

activities in the

discussion forum

18.73 8.125 .141 .251 .297

strategies are you using

to evaluate your

performance in the

discussion forum

18.92 8.394 .208 .207 .268

How are you going to

set you learning goals to

perform the task of the

assignment

19.96 7.878 .366 .570 .202

ONLINE LEARNING ASSESSMENT 55

In addition to use

rubric to assess your

performance on an line

discussion, what else

you will do

20.00 8.080 .413 .275 .203

In order to facilitate

team’s accomplishment

toward the goals, what

are the strategies

managers can provide

19.00 9.760 -.079 .426 .389

When you developed

the timeline of your

learning task, what is

the best approach to

start your timeline

19.12 8.506 .158 .501 .289

When you developed

the timeline of your

learning task, what is

the best approach to

start your timeline

18.38 9.606 .019 .320 .341

According to the above data, participants responded differently to the MCQ. The relationship among

responses indicated vast differences. Therefore, deleting items could increase the reliability of the test, but it is

ONLINE LEARNING ASSESSMENT 56

not necessary to delete items because the distance learners should have self-regulation skills. Revising items

would be much better. Adding items related to the construct will increase the reliability of the test.

Validity. In addition, to examine the validity, accuracy, and interpretation of the test scores, evidence

based on the internal structure will be used. Therefore, test homogeneity is obtained to measure the degree to

which different items measure the same construct of the test. It is obtained by calculating the coefficient alpha

for the test. Factor analysis shows the relationship and correlation among test items and determines the number

of factors and the kind of construct of the test, such as unidimensional or multidimensional, as in Table 2.

Table 2 shows that there is no consistency among the items of the test. The relationship is negative, so it is in

the opposite direction, and it does not measure one construct. Revising the blueprint of the test is required to

improve the reliability of the test.

According to the coefficient alpha, there are differences between the scores of test items. For example,

the Q1 is low >.5 (-.040), and the coefficient alphas of the rest are positive, but still low (.019) and (.366). This

shows that the items of the test might be measured within different constructs, or participants might have been

confused. Thus, examining the items that showed the low coefficient alpha led examiner to eliminate or revise

the items.

Component Matrix

Component

1 2 3 4

practices of setting

goals

.462 -.015 .633 -.267

ONLINE LEARNING ASSESSMENT 57

what skills should you

use to make

accomplishment on an

online learning within a

period of two months?

-.240 .232 .615 -.294

what criteria are you

looking for:

.575 .164 .091 .662

strategies are you using

to start your learning

activities in the

discussion forum

.261 .577 -.331 -.188

strategies are you using

to evaluate your

performance in the

discussion forum

.089 .560 -.311 .281

How are you going to

set you learning goals

to perform the task of

the assignment

.613 .545 .143 -.021

ONLINE LEARNING ASSESSMENT 58

In addition to use

rubric to assess your

performance on an line

discussion, what else

you will do

-.134 .691 .308 -.021

In order to facilitate

team’s accomplishment

toward the goals, what

are the strategies

managers can provide

-.676 .255 -.117 .194

When you developed

the timeline of your

learning task, what is

the best approach to

start your timeline

-.729 .466 .166 .073

When you developed

the timeline of your

learning task, what is

the best approach to

start your timeline

-.169 -.233 .519 .646

Result Performance Task

ONLINE LEARNING ASSESSMENT 59

The performance task involved writing a report that showed the participants’ self- assessment abilities.

The evaluation of the self-assessment task was based on a rubric. The result is shown in Table 4.

Table 4

NO Rubric Grade

1 20 Average

2 1 Poor

3 1 Poor

4 5 Poor

5 5 Poor

6 20 Average

7 5 Poor

8 10 Average

9 2 Poor

10 30 Good

11 10 Poor

12 5 Poor

The overall results showed that participants’ performances on the performance task were poor in terms

of determining the area of development, the quality of response, and the length of the report. Almost all

participants responded vague and uncertain responses, even though there were no responses provided. The

participants who responded to the performance task, they did not support their ideas with examples, and they

reflected only a few words. Participants who scored average performed better in terms of the length of the

report (a paragraph) and supported their ideas with a few examples.

ONLINE LEARNING ASSESSMENT 60

In conclusion, the data analysis of both MCQ and the performance task show variability among the

scores of the participants. Therefore, revising and improving the construct of the test is required in order to

provide the decision maker in the SCPTT an evidence for improving particular areas in the training materials

and practices of professional training.

Reflective Discussion

According to the results of the data analysis of the MCQ, the test is required for improvement from

different angels. The test will be developed with multidimensional constructs. Also, it will focus on three

factors related to SR: planning, reflection, and monitoring. Furthermore, the type of scale will be changed. The

scale will consist of a 7-point Likert scale: 1 (strongly agree) to 7 (strongly disagree). Moreover, the test will be

improved by developing the same number of response options for all items on the test. The results of the

current test identified the ineffectiveness of the test. Because it is a pilot test, it provides the examiner with

sufficient and effective knowledge and skills about developing online learning assessment. Moreover, it

informs the examiner about which skills distance learners must acquire to be successful in online learning. In

addition, as the test will be administered by the trainees in the SCPTT, the results of the test will add value to

the overall performance of professional training in the SCPTT. Specifically, learning activities and evaluation

strategies will be improved to help trainees master SR skills.

In addition, the SCPTT has an accreditation from the University of London. Thus, the teachers are

required to achieve certain standards of performance across the training programs. Therefore, adding items to

the test that focus on certain standards will support the process of accreditation.

Developing educational assessment paper must be done through systematic procedures including

developing, reviewing, administering, and also using information about educational programs undertaken for

the purpose of improving students’ learning performance and development. In fact, personally, I am new to

educational assessment field and I was unaware about the procedures of developing an assessment paper. I was

ONLINE LEARNING ASSESSMENT 61

struggling at the beginning of the course, and I decided to drop it. However, after I started reading the

materials and then pursued the classroom activities including discussion, reasoning, and sharing experiences,

then, the subject became easier. In addition, attending AERA annual meetings helped me gain deep knowledge

about the kind of assessments that could be developed related to my study area. For instance, TPACK

instrument (Technological Pedagogical Content Knowledge) can be used to assess teachers’ skills in evaluating

digital content. Furthermore, using the TPACK instrument it is possible to assess the differences in the

knowledge and skills between novice teachers and experienced teachers through taking a professional

development. Another lesson that I learned is to use an instrument called STAAR to assess the self-efficacy of

teachers in terms of personal efficacy and teaching efficacy. Readiness was observed in mathematics teachers

for using this technology in order to develop students’ performance in mathematics.

I faced a challenge in developing the assessment paper. The difficulty was occurred during analyzing

and discussing the data. That is because I have not taken the statistics and quantitative courses so, it was

difficult for me. Overall, it has been a good experience to practice the whole process of the test development.

Finally, the concept of developing measurement instruments is very clear and I understood the flaws of

the current constructs. Therefore, I will rely on the standards of education and psychology in developing the

items of the test. Also, I will use the Likert-point scale, and will study the content of the construct profoundly.

As I have enough background about the content of the test, I can develop a good and effective test.

In my opinion, developing an assessment paper is difficult for a new student, but it is beneficial to

experience the process of developing it. In addition, self-regulation is a new notion, so it was an excellent

opportunity for me to understand the background, theoretical framework, and studies that was done around it.

Moreover, I gained knowledge about the theory of SR and the application and implication of SR in online

learning environments. I am now confident in my ability to measure different topics related to educational

technology.

ONLINE LEARNING ASSESSMENT 62

Running head: ONLINE LEARNING ASSESSMENT 63

References

Brookhart, S. M., & Niko, A.J. (2015). Educational assessment of students. Upper Saddle River,

NJ: Pearson Education, Inc.

Johnson, R. B., & Christensen, L. (2014). Educational research: Quantitative, qualitative,

and mixed method. Los Angeles, LA: SAGE Publication, Inc.

Cook, D. A., & Steinert, Y. (2013). Online learning for faculty development: A review of the

literature. Medical Teacher, 35(11), 930-937. doi:10.3109/0142159X.2013.827328

Burnette, J. L., O'Boyle, E. H., VanEpps, E. M., Pollack, J. M., & Finkel, E. J. (2013). Mind-sets

matter: A meta-analytic review of implicit theories and self-regulation. Psychological

Bulletin, 139(3), 655-701. doi:10.1037/a0029531

Dabbagh, N., & Bannan-Ritland, B. (2005). Online learning: Concepts, strategies and application. Upper

Saddle River, NJ: Prentice Hall.

Dabbagh, N. & Kitsantas, A. (2012). Personal learning environments, social media, and self-

regulated learning: A natural formula for connecting formal and informal learning. The Internet and

Higher Education, 15(1), 3-8. doi:10.1016/j.iheduc.2011.06.002

World Bank. 2013. Main report. Washington DC: World Bank.

http://documents.worldbank.org/curated/en/2013/01/17406411/education-oman-drive-quality-vol-2-2-

overview

Ghosh, P., Jagdamba, R. S., Rashmi, P. J., & Singh, R. P. (2012). Towards more effective

ONLINE LEARNING ASSESSMENT 64

training programmes: A study of trainer attributes. Industrial and Commercial Training, 44(4), 194 –

202. doi: http://dx.doi.org.mutex.gmu.edu/10.1108/00197851211231469

Kearns, L. R. (2012). Student assessment in online learning: Challenges and effective

practices.  Journal of Online Learning and Teaching, 8(3), 198-208. Retrieved from

http://search.proquest.com.mutex.gmu.edu/docview/1500356017?accountid=14541

Kochtanek, T. R., & Hein, k. k. (2000). Creating and nurturing distributed asynchronous

learning environments. Online information review, 24(4), 280-294. Retrieved from

http://search.proquest.com.mutex.gmu.edu/docview/194493715?accountid=14541

Kwan, L. S. L., & Yunus, M. M. (2014). Group participation and interaction in ESL wiki

collaborative writing among malaysian gifted students. Asian Social Science,11(2),

59-68. doi:10.5539/ass.v11n2p59

Law, V., Ge, X. & Eseryel, D. (2016). The Development of a self-regulation in a collaborative

context scale. Springer Science+Business Media, 21, 1-12. doi: 10.1007/s10758-016-9274-z

MacDonald, J. (2003). Assessing online collaborative learning: Process and product. Computer

& Education, 40(4), 377–391. doi:10.1016/S0360-1315(02)00168-9

Miller, J.W., & Tucker, J. S. (2015). Addressing and assessing critical thinking in intercultural

contexts: Investigating the distance learning outcomes of military leaders. International Journal for

Intercultural Relations, 48, 120-136. doi:10.1016/j.ijintrel.2015.07.002

Milligan, A. T. & Buckenmeyer, J. A. (2008). Assessing Students for Online Learning.

ONLINE LEARNING ASSESSMENT 65

International Journal on E-Learning, 7(3), 449-461. Retrieved from

http://search.proquest.com.mutex.gmu.edu/docview/210337772?accountid=14541

Oncu, S., & Cakir, H. (2011). Research in online learning environments: Priorities and

methodologies. Computer and Education, 57(1), 1098–1108. doi:10.1016/j.compedu.2010.12.009

Redmond, B. F. (2106, Feb 20). Goal Setting Theory. Atlassian News. Retrieved from

https://wikispaces.psu.edu/display/PSYCH484/6.+Goal+Setting+Theory

Nicol, D.J. & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: A

model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218. doi:

10.1080/03075070600572090

Panadero, E, Tapia, J. A., Huertas, J. A. (2012). Rubrics and self-assessment scripts effects on

self-regulation, learning and self-efficacy in secondary education. Learning and Individual References,

22, 806-813. doi:10.1016/j.lindif.2012.04.007

Pritchard, R. D., Young, B. L. Koenig, N., Schmerling, D., & Dixon, N.W. (2012). New

development in goal setting and task performance. Florida, FL. Routledge.

Wells, A. Cartwright-Hatton, S. (2004). A short form of the metacognitions questionnaire:

Properties of the MCQ-30. Behavior Research and Therapy, 42, 385–396. doi:10.1016/S0005-

7967(03)00147-5

Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background,

ONLINE LEARNING ASSESSMENT 66

methodological developments, and future prospects. American Educational Research Journal, 45(1),

166–183. doi: 10.3102/0002831207312909

Running head: ASSESSING ONLINE LEARNING ENVIRONMENT 67