#pearsoncite

Let’s Launch a CBE Program: An A-Z Workshop

#pearsoncite

Paul Bowers

Strategic Consultant

Pearson

paul.bowers@pearson.com

@pebowers1

Joan Bouillon, PhD

Director Regulatory Compliance

Pearson

Joan.bouillon@pearson.com

Barbara Rowan, PhD

Director of Academic Research, Efficacy, and Evaluation

Pearson

barbara.rowan@pearson.com

Kate Kazin, J.D., PhD

Chief Academic Officer

Southern New Hampshire University

Kate.kazin@snhu.edu

#pearsoncite

Mary Adkison

Academic Strategist, Institutional Efficacy

Pearson

Mary.klingman-adkison@pearson.com

Elizabeth Adkins

Director of Instructional Design

Pearson

elizabeth.adkins@pearson.com

Russ Lewinter

Manager, Product Development

Pearson

Russ.lewinter@snhu.edu

#pearsoncite

Workshop Agenda

1. Getting Started- WHY CBE and Key Realities

2 Designing CBE Programs

3. Institutional Supports for CBE

4. Leading Change and Engaging Your Campus

#pearsoncite

Workshop Goals

• Engage Planning for CBE

• Identify Key Drivers and Purposes

• Focus on Essential Elements and Key Decisions

• Simplify Many Tentacles of CBE

• Develop Strategies to Engage Others

#pearsoncite

Useful Frameworks

• CBE Playbook

• C-BEN Design Elements for CBE

• C-RAC Accreditation Criteria

#pearsoncite

CBE PlaybookMaintaining coordination across 7 key workstreams

Summary

Our framework for the CBE Playbook – it provides the 7 key work streams necessary for CBE strategy.

Note on use

The CBE work streams are not sequential. Rather, they run in parallel with specific activities occurring within in each stream throughout the program development phase.

1. CBE Strategy & Integration 5.

Technology & Infrastructure

4. Student Success

2. Organization

3. Program

Development

6. Program

Management

7. Marketing &

Recruiting

#pearsoncite

Key functions across workstreams

Strategy & integration

Strategy development

Project management

Metrics & Accountability

Executive leadership

Financial modeling

External relations

Accreditation & approvals

Change managementO

rg.

Faculty Development

Curriculum D&D

Delivery Model D&D

Assessment D&DD

ev.

Orientation & Readiness

Retention strategy

Service Processes

PLA & Degree pathways

Succ

ess

Learning infrastructure

Reg & Assessment mgmt

Data & learning analytics

Retention & support solutions

Tech

Resource planning

StaffingVendor

relationsPartnerships

Tracking & reportingM

gmt

Marketing strategy

Lead genFunnel

managementRegistration &

advisingEM

#pearsoncite

C-BEN Program Design

Elements

How strongly do CBE programs agree the

following design elements are needed for a robust and healthy program?

#pearsoncite

C-RAC Accreditation Criteria

#pearsoncite

Why CBE?What Is CBE?

#pearsoncite

Thinking about Learning

Pick a time you learned something new.

Complete the handout.

#pearsoncite

In other words, CBE:

• Focuses on what students can do with what they know

• Assesses learning directly rather than relying on proxies (e.g., credit hours and seat time)*

• Allows multiple attempts to acquire mastery

• Does not define how the learning should be achieved

*But may require “translation” into credit hours for administrative or regulatory purposes

#pearsoncite

Why CBE?

• It’s the latest fad in higher education

• Your president or provost said you needed aCBE program

• Your IT team has too much time and too many resources

✗✗

#pearsoncite

The conventional transcript is a black box

=

#pearsoncite

Advantages for programs and institutions

• Clear evidence of what students know and can do

• Insights into student progress

• Easier alignment with industry – employers “speak” competency

#pearsoncite

Advantages for students:

• Flexibility (schedule and pace)• Transparency (progress and outcomes)• Lower cost (potentially)• Greater individual support for learning (usually)• Removal of irrelevant barriers (like time)• Greater relevance, especially for adult learners

#pearsoncite

Discussion Exercise:Tell us about YOUR Interests in CBE

• What is driving YOUR Institution/Thinking?

• What goals does a CBE program help you achieve?

• What are you most unsure about regarding CBE?

#pearsoncite

Key Realities:Accreditation and Financial Aid Eligibility

#pearsoncite

Compliance Agencies

• State agency approval

• Programmatic accreditor

• Licensing board

• Institutional accreditor

• Department of Education

#pearsoncite

Accreditation Requirements

• Competencies are externally referenced• Outcomes emphasize performance, not just

knowledge• Regular and substantive interaction with faculty• Competencies reflect achievement associated

with the degree level

#pearsoncite

• Good practices in assessment and measurement

• Validate quality through feedback

• Credit hour equivalency method

• Admission requirements and screening

• New student orientation

• Learning activities including externships

• Advanced standing policy

Accreditation Requirements

#pearsoncite

• Faculty roles and qualifications

• Grading policy – pass/fail, points, other

• Satisfactory academic progress

• Cancellation and refund policies

• Transcripts and student information system (SIS)

• Transfer and transition to traditional delivery plan

Accreditation Requirements

#pearsoncite

DOE Requirements

• “an institution must establish a methodology to reasonably equate the direct assessment program…to credit or clock hours”

• “a full-time student is expected to complete the equivalent of at least 24 semester or trimester credit hours…900 clock hours” in a minimum of 30 weeks

#pearsoncite

DOE Requirements

• “…process to document a student’s Enrollment Status”, i.e. full time, etc.

• In “any seven-day period…at least one day of educational activity occurs… educational activity in a direct assessment program includes regularly scheduled learning sessions”

#pearsoncite

DOE Requirements• Satisfactory Academic Progress (SAP) policy “must include

procedures to determine whether a student has completed sufficient competencies to complete the program within the maximum time frame…” and be “…based on the number of competencies associated with a specified number of credit hours, clock hours, or equivalencies…”

#pearsoncite

Discussion Exercise

• The follow slide has THREE accreditation issues you will need to address at some point

• We will divide into three small groups. • Each groups takes one question• Take 5-7 minutes to discuss how you would likely

respond• Report out to the whole what you concluded

#pearsoncite

Accreditation Application Items

1. What are the procedures to transition a CBE student to the institution’s traditional delivery program.

2. Describe the refund policies for the CBE program.

3. Provide policies on advanced standing, PLA, experiential learning, and/or transfer of credit for CBE programs.

#pearsoncite

Designing CBE Programs: Key Decisions

#pearsoncite

Exercise: Decisions, decisions

• Who are your stakeholders?– Internal – External (e.g., employers)

• What is your goal/purpose in offering CBE?• What’s the scope of your CBE initiative?

– Single course? (not recommended)– Program?– Institution or system?

• Credit-based or direct assessment?

@cathrael_kazin

#pearsoncite

Two options

• Start from scratch– Build new CBE from the

bottom up

• Retrofit – Try to “competency-ize”

existing programs”

@cathrael_kazin

#pearsoncite

Once that’s settled…

• You need a set of claimsabout what your graduates know and are able to do

• These are your competency statements

#pearsoncite

What makes a good competency statement?@cathrael_kazin

Measurable and

observable

Clear

Significant

Can stand alone

Reflects doing, not

just knowing

Has external meaning

#pearsoncite

Exercise: Evaluating Competency statements

Graduates of CITE University will be able to think critically

Understands the concept of supply and demand

Can analyze a problem and recommend solutions in writing

Is able to solve quadratic equations

Can name the three types of receptors in the human body

Conveys information by creating charts and graphs

#pearsoncite

Assessing Competencies: the Heart of CBE Programs

#pearsoncite

Assessment

• Driving force behind a strong CBE program

• Two-step Process– Assessment development and score validation

– Determining cut scores

#pearsoncite

Assessment Development

StandardsPsychometric

EvaluationValidity

#pearsoncite

Assessment Best Practices

• Each competency must be measured more than one time

• Each competency must be measured more than one way

• Assessments should focus on clusters of competencies as they might appear in real world

#pearsoncite

Assessment Types

Performance-based Assessment (real-world

focus)

Objective Measures of

Competencies

Diagnostic Tests of Knowledge

@BarbaraRowan

#pearsoncite

Assessments

• Assessment maps to competencies

• Assessment measures a student’s knowledge and skills AND how well the student can integrate, synthesize, and use the knowledge and skills necessary to become a community of practice

#pearsoncite

Assessments

The more “real world” the assessment is, the more value the results hold for determining a student’s mastery of the competencies.

#pearsoncite

Psychometric Evaluation

• Reliability – indicates consistency of test scores

• Item Analysis – indicates items that are performing poorly

• Rubric Evaluation – to ensure that the rubric is properly evaluating subjective measure

@BarbaraRowan

#pearsoncite

Score Validation

Test Content

Relations to Other

Variables

Response Processes

Test Consequences

Internal Test Structure

@BarbaraRowan

#pearsoncite

Framework for Determining Mastery

• Must determine what separates masters from non-masters

• Validity evidence required to support decisions

@BarbaraRowan

#pearsoncite

Cut Scores – Questions to Ask

What do Students Need to Know and Be Able to Do at Each Level?

What Performance Levels will be

Reported?

Are Cut Scores Necessary?

@BarbaraRowan

#pearsoncite

Cut Score Options

@BarbaraRowan

#pearsoncite

Five Basic Steps

Select Judges

Define borderline knowledge and skills

Train the judges

Collect judgments

Choose a cut score

by combining judgments

@BarbaraRowan

#pearsoncite

Choosing a Method

• No one best method

• Choice should depend on what kind of judgments can be gotten and believed

• Based on types of questions in the test (multiple choice; constructed-response; rubric-based

@BarbaraRowan

#pearsoncite

NAEP Suggested Levels

Basic Level

• Partial Mastery of knowledge and skills

Proficient Level

• Solid Academic Performance and Demonstrated Competence over Challenging Subject Matter

Advanced Level

• Signifies Superior Performance on Challenging Subject Matter

@BarbaraRowan

#pearsoncite

Cut Score Process

@BarbaraRowan

Set Provisional Cut Scores

Establish Operational Cut Scores

Document the ProcessEvaluate the Results of

Using the Cut Scores

#pearsoncite

Provisional Cut Scores

• Determined by people who are aware of what students should know and be able to do

• Determine how many score points students have to earn to demonstrate they have reached the level of knowledge and skill for each level

@BarbaraRowan

#pearsoncite

Provisional Cut Scores

• Choose Cut Score Method

Step 1

• Select Judges, that is, 5-15 subject matter experts

Step 2• Train Judges in

the Cut Score Method Chosen

Step 3

@BarbaraRowan

#pearsoncite

Operational Cut Scores

@BarbaraRowan

Is it better to pass a student who should fail?

Or fail a student who should pass?

#pearsoncite

Operational Cut Scores

Test Score Fluctuations

Pass

Fail

@BarbaraRowan

#pearsoncite

Document the Process

@BarbaraRowanRationale

Qualifications

Training

Steps

Data shared with Judges

Judges’ Ratings

Evaluation Forms from

Judges

Rationale for Adjustments

#pearsoncite

Evaluating the Results of Using Cut Scores

• Occurs after at least one round of testing and reporting of results has occurred

• Requires the validation of cut scores

• Ongoing process

@BarbaraRowan

#pearsoncite

Validity Study

• Are the results reasonable?

• Are failure rates far higher or lower than expected?

• Are outstanding students failing to reach the highest level?

• Are poorly prepared students reaching the highest level?

@BarbaraRowan

#pearsoncite

Discussion

• What types of assessments do you envision for YOUR CBE program?

• How would you envision addressing the development of assessments – who should be involved?

#pearsoncite

Developing The Institutional Support Model

#pearsoncite

Three Program Models

#pearsoncite

ProgramManagement

#pearsoncite

Program Model

#pearsoncite

Student Support Model

College for America

Academic planSelf directed student progression through competencies

Source: Adapted from EDUCAUSE Next Gen Grant Recipient Profile

CoachSet and monitor

goals

Online networksPeer support

Learning PartnerSupport and encourage

MentorEmployer /

community member

#pearsoncite

Kentucky Virtual Technical College Model

Source: Adapted from EDUCAUSE Next Gen Grant Recipient Profile

Pre-assessment Activity Post-assessment

Learn on demand course design: pre-assessment allows path to test out

FacultyLead each

course, provide

feedback

Student support

staffIntensive support

#pearsoncite

Texas Affordable Baccalaureate Program

Source: Adapted from EDUCAUSE Next Gen Grant Recipient Profile

Content ExpertsDevelop content

materials

CoachesAssigned

enrollment to graduation

TutorsAssigned to

content domains

FacultyFace-to-face &

online

Lower-division coursework Upper-division coursework

Oversight teamFaculty & Business leaders develop competencies

#pearsoncite

Exercise: Evaluating Competency statements

Graduates of CITE University will be able to think critically

Understands the concept of supply and demand

Can analyze a problem and recommend solutions in writing

Is able to solve quadratic equations

Can name the three types of receptors in the human body

Conveys information by creating charts and graphs

#pearsoncite

Key Questions

• What is the role of faculty?

• What is the role of staff?

• How are activities and information shared between the two?

#pearsoncite

Discussion Exercise

• In Small Groups:– Identify KEY Constraints for Support Model and Roles

– What would be the Key Characteristics of YOUR Support Model?

#pearsoncite

Data Informed Learning Design

#pearsoncite

Data Informed

Learning Design

DIGITAL LEARNING ENVIRONMENT

INFORMED DECISIONS

DATA

Pearson Digital Footprint

INFORMED DECISIONS

PREDICTIVE ANALYTICS

Enterprise-level: Programs and courses across the entire institution

Program-level:Departments and/or Programs (e.g. CBE)

Course-level:Course(s) taught by

individual instructors

Student-Level: Student

behavior & performance

Data Informed Learning Design

INFORMED DECISIONS

Analysis

Strategy + Action

Measurement

Data Modeling

& Predictive

Alerts

Learning Design

Professional Development

Launch

Monitor

Measure

Iterate

Actionable Data Dashboard

INFORMED DECISIONS

PREDICTIVE ANALYTICS

#pearsoncite

Discussion Exercise

• What are the MOST important things you would like to know about your learners?

#pearsoncite

Conversation Roundtables

#pearsoncite

Closing Thoughts:Change Leadership Strategies

#pearsoncite

Change Leadership

• Getting stakeholders to buy into

change

• Ensuring the process stays under

control

• Minimizing risk

• Keeping the initiative on budget

• Articulates a vision for the

future

• Mobilizes and coordinates

resources necessary for the

change

• Creates engines for accelerating

innovation

Change LeadershipChange Management

#pearsoncite

Change Leadership Strategies •Involve key stakeholders as early as possible

•Create faculty innovation models

•Clearly articulate goals and outcomes -WHY CBE?

•Empower key stakeholders through distributed problem-solving and

iterative design

•Engage employers and other external stakeholders in meaningful ways

•Understand key constraints and limits–Which can you overcome and which do you need to live with?

#pearsoncite

Change Leadership Examples

• U Texas Center for Transformational Learning

• Purdue IMPACT Program

• Edmonds Community College

#pearsoncite

Final Question

• What are your greatest concerns about change?

• What will YOU do to lead change on your campus?