Quality: Becoming what you measure

Chère Campbell GibsonPresented at the 17th Annual

TelecoopTechnology Conference

The inspiration

Dunning, B., Van Kekerix, M., & Zaborowski, L.

(1993). Reaching learners through tele-

communications. San Francisco, CA: Jossey-

Bass Publishers.

Our focus

Quality standards over time Core quality criteria for e-learning Benchmarking quality criteria Ethics and values in quality assessment

Old measures of quality

“The historic pattern in post-secondary

education has been to weigh the quality of non-

traditional education by how closely it mirrors

the format and content of education in a

traditional classroom.”Reaching learners through telecommunications, (1993), (p. 187)

Later assessments of quality

Enrollment - number of learners, often with an emphasis on the dollars generated

Attrition – number of students who drop out of the program they enrolled in

leading to research on learner characteristics

and later to program characteristics or

organizational obligations

Organizational obligations

Organizational obligations, i.e., program

features deemed important for program quality

emerged in the 1990s and addressed distance

education in general.

Organizations involved

WICHE American Council on Education Pew Foundation Middle States Association of Schools and

Colleges American Distance Education Consortium

Common features

Curriculum Faculty support Learner support Technology support & infrastructure Institutional support Evaluation Cost

Later developments

Began to focus more specifically on the online

environment including efforts of such groups

as: WCET Sloan Foundation Institute for Higher Education Policy (IHEP)

Common features

Curriculum/Course development Teaching/Learning (more overt emphasis) Faculty support Learner support Course structure Institutional support Evaluation/Assessment Cost

A question emerges

But one might ask,

What are the hallmarks of quality

in e-learning?

Core quality criteria emerge

Two studies in the last five years explored core

quality criteria for online learning

Both began with literature reviews, adding

Case studies – 6 institutions (IHEP)

Delphi - 54 distance education experts

(Mariasingam)

Both studies resulted in 24 core criteria – 75%

in common

Institutional support

A documented technology plan that includes electronic security measures is in place and operational.

The reliability of the technology delivery system is as failsafe as possible.

A centralized system provides support for building and maintaining the distance education infrastructure.

Course development

Instructional materials are reviewed periodically to ensure they meet program standards.

Courses are designed to require students to engage in analysis, synthesis, and evaluation as part if their course and program requirements.

Teaching/learning

Student interaction with faculty and other students in an essential characteristic and is facilitated in a variety of ways, including voice

mail and/or email. Feedback to students on assignments and

questions is constructive and provided in a timely manner.

Course structure

Before starting an online program, students are advised about the program to determine if they possess the self-motivation and commitment to learn at a distance, and

If they have access to the minimal technology required by the course design.

Course structure con’t

Students are provided with supplemental course information that outlines course objectives, key concepts and ideas, learning outcomes summarized in a clearly written statement.

Students have access to sufficient library resources that may include a virtual library…

Student support

Students receive information about programs including admission requirements, tuition and fees, books and supplies, technical and proctoring requirements and student support services.

Throughout the duration of the course or program, students have access to technical assistance including detailed instruction…

Student support, con’t

Questions directed to student service personnel are answered accurately and quickly with a structured system in place to address complaints.

Faculty support

Technical assistance in course development is available & faculty are encouraged to use it

Faculty members are assisted in the transition from classroom teaching to online instruction and are assessed during the process.

Instructor training/assistance …continues throughout the progression of the course.

Evaluation and assessment

The program’s educational effectiveness and teaching/learning processes are assessed through an evaluation process that uses several methods and applies specific standards.

Data on enrollment, costs and successful/

innovative uses of technology are used to evaluation program effectiveness.

Evaluation & assessment con’t

Intended learning outcomes are reviewed regularly to ensure clarity, utility and appropriateness.

Areas of disagreement

Prescriptive teaching practices, both in terms of process and content

Supportive environment, including faculty incentives, teaching awards, etc.

Ongoing quality assurance/TQM Involvement of stakeholders, especially

employers, in assessment.

Missing feature

Benchmarks - A reference point serving as a

standard for comparing or judging other things

New International Webster’s Concise Dictionary of the English Language (2000)

Examples

A core criterion:

Before beginning an online program students

are advised about the program to determine if

they possess the self-motivation and

commitment to learn at a distance.

Benchmarks

Students are provided written documents that help them determine their suitability, including motivation and commitment - MINIMUM

Students are required to conduct an online self-analysis and discuss results in a personal advisory session with student counselor or academic advisor - BEST

Next steps

We need to: Determine which quality criteria we feel are

appropriate for our education and training

programs and Establish benchmarks for those criteria

Our successes

As a field we have: Moved beyond acceptance of marginal

standards; Identified core quality criteria; Recognized the absence of quality

benchmarks in the truest sense of the word;

BUT

A question

Do the criteria found in the literature and

presented by a variety of agencies and

organizations fit the values of your institution?

Beyond functional quality

Ethical outcomes as a measure of quality Equity Climate Autonomy Relationships Others that fit your organizational values and

purposes

An easy example

“Before beginning an online program students

are advised about the program to determine if

they …have access to the minimal technology

required by the course design.”

What it doesn’t add is “If not, the institution will

ensure access to appropriate technology …”

Equity

Distance education provides access but for

whom? Communities of color Lower income individuals Persons with disabilities Geographically distant learners Speakers of English as a second language Those with limited educational background

Climate

Are we creating a positive environment for

learning?

Are the interactions among and between the

teachers, learners, content, the organization,

the technology that unites all conducive to

learning?

Autonomy

“Another potential measure of quality is the

degree to which programming supports and

advances differences and self-directedness

among learners.”Reaching learners through telecommunications (1993), p.

202

Relationships

Learner as customer – the buyer-seller

relationship

versus

Relationships built on meeting unique needs and mutual concerns and desires,

recognizing neglected constituencies and

striving to overcome barriers to participation

A truism

What a manager chooses NOT to measure

indicates which outcomes are less valued.

And you become what you measure!

And so you must ask

What have I failed to value through my

quality assessment?

It says a lot to others, including your learners!

Thank you

Chère Campbell Gibson

ccgibson@wisc.edu