Moving from “buy-in” to “process improvement” and to “planned change”

35
Moving from “buy-in” to “process improvement” and to “planned change” Jack Phadungtin & Darla Calvet National University, La Jolla, CA

description

Moving from “buy-in” to “process improvement” and to “planned change”. Jack Phadungtin & Darla Calvet National University, La Jolla, CA. About National University. Type: Private, Not-for-Profit Campuses: 28 Students: 27,000 Faculty: 3,100 (full-time & adjunct) - PowerPoint PPT Presentation

Transcript of Moving from “buy-in” to “process improvement” and to “planned change”

Page 1: Moving from “buy-in” to “process improvement” and to “planned change”

Moving from “buy-in” to “process improvement” and to “planned change”Jack Phadungtin & Darla CalvetNational University, La Jolla, CA

Page 2: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOAbout National University

Type: Private, Not-for-ProfitCampuses: 28Students: 27,000Faculty: 3,100 (full-time & adjunct)Degree programs: 98 (grad & undg)Classes: 800-1000/monthNewly admitted students: 17,000/YrIR & Assessment staff: 5

Page 3: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOMomentum to SuccessMomentum to Success

ContinuousImprovement

Planned Change

ProcessImprovement

Buy-In

Institutional Effectiveness

Page 4: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOIngredients of Winning Planned Change

Assessment Infrastructure Support from Administration Capability of the IR Office and Faculty

Assessment Process Assessment Cycle Governance bodies and their responsibilities

Effective Use of Assessment Results

Page 5: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOIngredients of Winning Planned Change

Assessment Infrastructure Support from Administration Capability of the IR Office and Faculty

Assessment Process Assessment Cycle Governance bodies and their responsibilities

Effective Use of Assessment Results

Buy

In

Page 6: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOIngredients of Winning Planned Change

Assessment Infrastructure Support from Administration Capability of the IR Office and Faculty

Assessment Process Assessment Cycle Governance bodies and their responsibilities

Effective Use of Assessment ResultsLess

Res

ista

nce

to C

hang

e

Page 7: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOAssessment Goals at NU

Enhance assessment qualityImprove assessment process

Utilize assessment results

Page 8: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOHow To Initiate Planned Change

Administration Faculty

Who Originates Change?

Page 9: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOHow To Initiate Planned Change

Administration Faculty

Office of IR & Assessment

Page 10: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOHow To Initiate Planned Change

Administration Faculty

Office of IR & Assessment

ChangePersonnel, Structure, Mind

ChangePersonnel, Structure, Mind

Page 11: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOGetting Started

Stop the “Resistance-to-Change” talk Isolate Problems & Prioritize Tasks

1. Assessment Cycle 2. The needed “buy-in”

– Support from Administration» Capability of the IR Office

– Support from Faculty» Understanding Benefit of Assessment

3. Governance bodies and their responsibilities

Page 12: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOBuy the “Buy-In”

Administration Cost and Benefit Analyses Recommendations Successful Pilot Projects

Faculty Remove Barriers Simplify Tasks Identify Project Champion Compensation

Page 13: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOBuy the “Buy-In”

Amplify Benefits of ChangePromote Effective CommunicationUnderstand Each Functions Involved in the

Process

Sample??

Page 14: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOAssessment Time Table (No longer used)

Page 15: Moving from “buy-in” to “process improvement” and to “planned change”

LOGORevised Assessment Cycle

Page 16: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOMoving from Buy-In to Process Planning

Create Master PlansIdentify Key Indicators/Measures

Academic Non-Academic

Plan for Data Collection and AnalysisEstablish practices of how results will be

used

Page 17: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOIdentify Key Indicators - Academic

Student Surveys

Page 18: Moving from “buy-in” to “process improvement” and to “planned change”

LOGO

Non-Academic Assessment Business Quality Survey Staff Survey

Identify Key Indicators – Non-Academic

Page 19: Moving from “buy-in” to “process improvement” and to “planned change”

LOGONon-Academic Measure 2008

Frequency of Survey – 4/Year 2 Quality of Business Survey (QBS) 2 HR Surveys

Jan

Pre

Feb Mar

QB

S

May Jun

Pos

t

JulQ

BS

–Foc

used

Aug

Pre

Sep Oct Nov

Pos

t

DecApr

Pre Post

Control Group 1 Control Group 2

Page 20: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOUsing Results to Drive Improvement

To Promote Positive Planned ChangeGive Credit to Project Originator/OwnerShare Results With StakeholderKeep Track of How Results Are UsedPrevent Reprimand for Inability to Achieve

the Set Goals

Page 21: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOInformation Optimization

Page 22: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOTracking Information Usage

Page 23: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOPromote Use of Evidence for Process and Structure ChangesChange Process to Support Change

Simplify and Minimize Steps Emphasize Quality Control

Change Structure to Support Change Enhance Institutional Capability

Page 24: Moving from “buy-in” to “process improvement” and to “planned change”

LOGO

Process Improvement2006

2008

Page 25: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOProcess / Structure Improvement:Academic Program Reviews

2006

2008

Page 26: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOProcess & Assign Responsibilities:Academic Program Reviews

Program LeadFaculty

UniversityAssessment

Council

Department Chair

Provost

School Assessment Committee (SAC)Members are appointed by the school Dean

School Dean

Undergraduate Council

President

Process Coordination

Process Coordination

Process Coordination

Graduate Council

Process Coordination

Process Coordination

Page 27: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOStructure / Capability Improvement

2006

2008

Page 28: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOImprovement Results

Academic Improvement 2007

Page 29: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOMAT EXIT SURVEY: Fiscal Year 2006 Graduates

ABSTRACTWithin the context of an academic program review at National University a

web-based survey was conducted asking recent graduates of the Masters of Arts in Teaching (MAT) program about the value of core MAT courses and whether they believed program outcomes were met. The survey was the second annual exit survey administered. Respondents rated the value of core courses, and rated agreement about whether program outcomes were met. Respondents were also given opportunity to provide open-ended rationale for their ratings.

PURPOSEThe purpose of the study was to discover, through perceptions and experiences

of recent graduates of the Masters of Arts in Teaching (MAT) program, the value of core MAT courses and whether graduates believed MAT program outcomes were achieved. Data was gathered by way of a web-based survey.

LITERATURE REVIEW

The purpose of an internal program review is to provide opportunity for continuous improvement through a systematic and cyclical process in which strengths of a program and areas for growth are identified (Kornuta, 2007; Daniels, 2005; LSU Planning & Review, 2006). Data for program reviews should come from multiple sources (NCATE, 2003; SIEU, 2006), and involve multiple measures such as teacher candidates' grades, scores on standardized entrance and exit exams and portfolios; community service data; employer surveys; and the extent to which students are meeting learning outcomes (Kornuta, 2007; Hess, 2002; University of Delaware, 2006).

Outcomes assessment allows institutions to measure goals unique to particular programs and identify successes and the need for growth without regard to the size or prominence of the institution (Jones, 1996; NCATE, 2003; SIEU, 2006; Western Association of Schools and Colleges, 2001).

BACKGROUNDAcademic programs are reviewed at the University on a five year cycle. The

comprehensive review process addresses the adequacy, viability, and necessity of the programs. During the program review process, a cross-check occurs to ensure that program goals and learning outcomes are aligned with the mission of the University, the mission of the School of Education, and requirements of both national and state accrediting agencies.

Within this context, a second exit survey was administered to recent graduates of the Masters of Art in Teaching (MAT) program. The data collected is compared to the previous survey, added to longitudinal data, and will contribute to the MAT Program Review.

Ron Germaine, Ed.D.Contributors: Jack Phadungtin, DBA; Cynthia Schubert, Ed.D.

DATA ANALYSIS

Population = 1648 N=290 (18% response rate) Not all respondents answered all questions; however, reliability is strong because respondents were representative of the geographic and online distribution of students.

Response to Question:“Rank the value of each of the six MAT core courses in

terms of how valuable each course was to the development of the your teaching ability.”

For additional information please contact:Ron Germaine, Teacher Education Department, National [email protected]

METHODOLOGY• A web-based survey was distributed in August 2006 to all students who

completed the capstone course MAT 640 between July, 2005 through June, 2006.

• Participants were provided with a unique personal identification number (PIN) code.

• Respondents were asked to use a Likert-scale to rank the value of each of the six MAT core courses in terms of how valuable each course was to the development of the their teaching ability;

• Respondents were asked to rank how strongly they agreed that the MAT Program helped to develop their knowledge, skills, and abilities in regard to each of the learning outcomes.

• The order of individual questions in each section was randomized to minimize the risk of survey fatigue occurring and contributing to skewed results.

• Validity of the survey was addressed by composing questions based on course learning outcomes and MAT program goals.

MAT CORE COURSE

# of + Resp.(# of +

Comments)

# of – Resp.(# of -

Comments)

Ratio+ : - Resp(+ : - com.)

% of +

Resp.

1. EDT 612: Creating Meaningful Learning with Technology

147(140)

29(25)

5:1(6:1)

85

2. MAT 641: Cultural Democracy: Contemporary, Local, and Global Issues

120(111)

38(33)

3:1(3:1)

76

3. MAT 642: Program Design: Curriculum Theory, Design, and Assessment

138(124)

25(25)

6:1(5:1)

85

4. MAT 643: Models of Teaching, Theories, Applications, and Practice

148(134)

16(15)

9:1(11:1)

90

5. MAT 644: Foundations and Principles of Curriculum

139(125)

27(26)

5:1(5:1)

84

7. MAT 640: Applications of Research for the Art of Teaching

183(171)

26(27)

7:1(6:1)

88

Average 146

(134) 20

(25) 7:1

(7:1) 85

Themes from open-ended responses: “Rank the value of each of the six MAT core courses in

terms of how valuable each course was to the development of the your teaching ability.”

Positive response themes (Note: Positive responses to the value of MAT core courses outnumbered negative responses by ~ 7:1)

• Content was relevant • Example comment: “Information presented correlated with real life instruction.”

• Instructors/instruction was good• Example comment: “Professor very accessible and responsive.”

Negative response themes (Note: Negative responses to the value of MAT core courses were outnumbered by positive responses by ~ 7:1)

• Content was not meaningful• Example comment: “We taught the course by presenting chapters. Few practical

connections to the classroom were made.”• Poor instructor/instruction

• Example comment: “Neither the learning outcomes nor the teacher's expectations were clear.”

Recommendations from Findings • Individually, we (faculty) need to reflect on our own teaching to ask whether we are

modeling best teaching practices and providing clear emphasis on relevant course and program learning outcomes.

• Supervision and professional development of adjunct instructors should take place annually for the purpose of enhancing the quality of teaching and instruction.

• For further study: Comments specific to each course should be reviewed by faculty who teach the course to look for ways to enhance student learning.

MAT Program Learning Outcomes: The MAT Program built my competency to…

# of + Resp.(# of Pos.

Comments)

# of – Resp.(# of Neg.

Comments)

Ratio+ : - Resp.(+ : - com)

% of +

Resp.

1. Reflect on my own teaching practice.

202(184)

8(8)

25:1(23:1)

96

2. Build professional relationships.

179(164)

31(31)

6:1 (5:1) 85

3. Build learning communities within my school and/or classroom.

190(168)

20(20)

9:1(8:1)

90

4. Analyze cross-cultural educational issues.

193(175)

17(16)

11:1(11:1)

91

5. Identify, describe, and apply theories of curriculum as they relate to State approved standards.

184(157)

26(25)

7:1(6:1)

88

6. Implement standards-based assessment in my educational setting.

185(157)

25(23)

7:1(6:1)

88

7. Use technology in my teaching.

187(172)

23(23)

8:1(7:1)

89

8. Use multiple teaching models.

196(184)

14(14)

14:1(13:1)

93

9. Implement, assess and evaluate standards-based curriculum.

182(156)

28(25)

7:1(6:1)

87

10. Use multiple assessment strategies

189(159)

21(21)

9:1(8:1)

90

11. Conduct action research in my teaching

194(163)

16(12)

12:1(14:1)

92

Average Number of Responses & Response Rate

189(167)

20(20)

9:1(8:1)

90

Themes from Open-Ended Responses:“Rank how strongly you agree that the MAT Program

helped to develop your knowledge, skills, and abilities in regard to each of the learning outcomes.”

Positive response to themes: (Note: Positive responses to program goals being met outnumbered negative responses by ~ 8:1)•The learning outcomes were met.

• Example statements: • “I had ample opportunity to effectively put into practice the learning

outcomes that were discussed in my masters classes”• “The entire program added to my capacity to implement State Standards.”• “The courses/professors contributed significantly to my ability to effectively

implement, assess, and evaluate the curriculum currently in use.”

Negative response themes: (Note: Negative responses to program goals not being met were outnumbered by positive responses by ~ 8:1)•Content was not meaningful; Courses are too short

• Example statements: • “One month is just not enough time to absorb and really develop these

learning outcomes”… ; “The course is not long enough…”•Greater need for emphasis on practical application of theory

• Example statement: “The program does not go far enough to take the learning out of books and into the classroom.”

Response to Question:“Rank how strongly you agree that the MAT Program

helped to develop your knowledge, skills, and abilities in regard to each of the learning outcomes.”

Example of Descriptive Statistics Used to Organize Data

Example of Descriptive Statistics Used to Organize Data

0 20 40 60 80 100 120 140 160 180 200 220

The MAT program developed my knowledge skills and abilities to:

Reflect on my teaching practice.

202 or 96%

8 or 4%

# of Students

Strongly Disagree

Disagree

Strongly Agree

Agree

Total # respondents to the question:

210

ReferencesDaniels, M. (2005). Student affairs program review, Ohio State University. Retrieved March 13, 2007 from

http://studentaffairs.osu.edu/programreview/static/Program_Review_Overview.pdfHess, F. (2002). The accreditation game. Education Next. Retrieved March 19, 2007 from

http://www.thefreelibrary.com/_/print/PrintArticle.aspx?id=91821095Jones, E. A. (1996). National and state policies affecting learning expectations. In M. Kramer (Series Ed.) & E.

A. Jones (Volume Ed.), New directions for higher education: Vol. 24. Preparing competent college graduates: Setting new and higher expectations for student learning. (pp. 7-17). San Francisco: Jossey-Bass.

Kornuta, H. (2007). Program review handbook: California Lutheran University. Retrieved March 19, 2007 from http://www.callutheran.edu/assessment/cycle/documents/ProgramReviewHandbookJan102007.pdf

LSU planning and review. (2006). Retrieved March 13, 2007 from http://appl003.lsu.edu/acadaff/lsustrategicres.nsf/$Content/Program+Review?OpenDocument#purpose

National Council for Accreditation of Teacher Education. (2001). Professional standards for the accreditation of schools, colleges, and departments of education. Washington, DC: NCATE.

SIEU. (2006). Report of the University Committee on Assessment: A proposed plan for assessment of undergraduate education at Southern Illinois University at Edwardsville. Retrieved March 13, 2007 from http://www.siue.edu/assessment/plan.html#Principle

University of Delaware. (2006). Periodic review report, p. 67 Retrieved March 13, 2007 from http://www.udel.edu/provost/FinalPRRMay2006.pdf

Western Association of Schools and Colleges. (2001). WASC handbook of accreditation. Alameda, CA: WASC.

“Rank the value of MAT640, Application of Research for the Art ofTeaching, in terms of how valuable the course was to the development

of your teaching ability.”

“The MAT program developed my knowledge, skills, and abilities to reflect on my teaching practice.”

Recommendations from Findings• Learning outcomes for each course should be reviewed for clarity, precision,

relevance, and alignment with program learning outcomes.• Learning outcomes, assignments, and evaluation for each course should be reviewed

for alignment with appropriate program learning outcomes.

0 20 40 60 80 100 120 140 160 180 200 220

MAT 640:Applications of Research for the Art of Teaching

How valuable was this course to the development of your teaching ability?

183 or 88%

26 or 12%

# of Students

209

Not at all Valuable

Not too Valuable

Very Valuable

Somewhat Valuable

Total # respondents to the question:

Page 30: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOMomentum of Improvement

Minimize Resistance to Change Create Successful Pilot Projects “Snow Ball” Effect

Learn from MistakesProvide Continuing Support

Customized HelpRewards and Pressure

Positive Reinforcement Peer Pressure

Page 31: Moving from “buy-in” to “process improvement” and to “planned change”

LOGO

Let’s Recap

Page 32: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOGetting Started

Stop the “Resistance-to-Change” talk Isolate Problems & Prioritize Tasks

1. Assessment Cycle 2. The needed “buy-in”

– Support from Administration» Capability of the IR Office

– Support from Faculty» Understanding Benefit of Assessment

3. Governance bodies and their responsibilities

Page 33: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOGetting Started

Stop the “Resistance-to-Change” talk Isolate Problems & Prioritize Tasks

1. Assessment Cycle 2. The needed “buy-in”

– Support from Administration» Capability of the IR Office

– Support from Faculty» Understanding Benefit of Assessment

3. Governance bodies and their responsibilities

Page 34: Moving from “buy-in” to “process improvement” and to “planned change”

LOGOGetting Started

Stop the “Resistance-to-Change” talk Isolate Problems & Prioritize Tasks

1. Assessment Cycle 2. The needed “buy-in”

– Support from Administration» Capability of the IR Office

– Support from Faculty» Understanding Benefit of Assessment

3. Governance bodies and their responsibilities

DON’T GIVE UP

Page 35: Moving from “buy-in” to “process improvement” and to “planned change”

National University