2015 Pre-Examiner 2015 Pre-Examiner Training and Training and

Preparation CoursePreparation Course

The Journey Begins (Continues)!

Before We Begin….Before We Begin….

Parking Parking LotLot

Logistics•Safety•Location of exits•Use of phone (please place on stun)•Restrooms•Parking Lot

Complete Training Evaluation Forms AND

Plus/Delta

Quality Texas Meeting Quality Texas Meeting BehaviorsBehaviors

• Be respectful

• Share openly

• Have only one conversation at a time

• Mute phones

• Return from breaks on time

• Take care of personal needs

• Clean up after yourself

• Stay on point

• Work toward consensus

• Understand that silence means affirmation

• Employ “ELMO” as needed

• Have fun

Ground Rules

Learning Outcomes Gain an understanding of the

Framework; Review

Apply the Six-Step Independent Review Process to an Award Application;

Choosing Key Factors, Strengths/OFIs, Scoring, and Key Themes

Write Better Feedback Comments Using the Comment Guidelines

Improve your BRAND!

Course Overview Tuesday – Quick review of the

Baldrige Framework; Review the Organizational Profile; Key Factors; Discuss Six-Step IR Process; Review Items 1.1

Wednesday – Six-Step IR Process; Items 2.1 and 5.2

Thursday – Six-Step IR Process; Review 7.1 and 7.5; Scorebook Navigator Class

Introductions – Introductions – At At Your TableYour Table

• Who you are!– Name / City

• What you do!– Organization– Job title

• Experience with QTF or Baldrige Criteria

• Your expectations of this Training

About Quality Texas

Laying Concrete

Quality Texas Foundation

Mr. Ryan Gonzales, Director of Operations

ryangonzales@quality-texas.org; (512) 940-8282

Ms. Lin Wrinkle, Director of Administration

linwrinkle@quality-texas.org; (512) 818-3901

Dr. Mac McGuire, Chief Executive Officer

drmac@quality-texas.org; (512) 656-8946

201 Woodland Park, Georgetown, Texas 78633-2007

www.quality-texas.org

Key Facts About Quality Texas Foundation

President Reagan concept 1987-1989; Mac Baldrige - Secretary of Commerce; died in rodeo accident

State Concept 1990; Started 1992; Endorsed by then Governor Ann Richards; November 1992 huge kickoff in Houston

Founded in 1994 as a 501(c)3 not-for-profit corporation

Full-time staff of two people; one person part time; around 300 state-wide volunteers

Key Facts About Quality Texas Foundation

Funded through Memberships and Sponsorships, Customized training workshops, Performance Excellence Program participant fees,

Annual Texas Quest Conference

Recognized as the most effective state program, with more Baldrige recipients than any other state (19); 52 state winners

QTF’s Vision and MissionQTF’s Vision and MissionVision: The Quality Texas Foundation, the preeminent state program, will continue to lead the way nationally in innovation by establishing Communities of Excellence (CoE) throughout the state.

Mission: QTF exists to assist individuals and organizations in their continuous improvement efforts thereby positively impacting our communities, state, and nation.  

Services We Provide• Assessment; Feedback. Recognition and Awards

• Examiner Training (You are here!)

• Applicant Training (how to apply; how to write; how to assess; site visit training; COE)

• Training and Coaching Solutions

– Customized training and coaching based on customer needs within their organization

– Public training workshops

• Membership Levels w/various benefits per level to

include Corporate/Individual options with discounts

Why Organizations Apply• Outside evaluation of organizational goals, objectives, and values

• Improve financials and employee engagement

• Helps build a common, holistic, and systematic view of the organization

• Framework provides a common language and standardized method to examine processes and performance; Receive objective feedback

• Tracks progress – Get to the “next level” of performance

• Fosters benchmarking within and across industries

ASSESSMENT PROCESS

RESOURCES

NEXT STEPS

• FEEDBACK REPORT FOR JUDGES (Award) & APPLICANT (all levels)

• RECOMMENDATION TO BOARD OF DIRECTORS (Award Level only)

• JUDGES REVIEW (Award)

• BOARD OF DIRECTORS DECISIONS (Award)

• FEEDBACK TO APPLICANT (all levels)

• ANNUAL AWARDS CEREMONY (all levels)

• Criteria Books• Application•QTF Website•QTF Staff

CORE VALUES

• Systems Perspective• Visionary Leadership• Customer-focused excellence• Valuing People• Organization Learning & Agility• Focus on success

Indepe

ndent

Review

Consen

sus

Site V

isit

• Score• ID Key Factors Strengths & OFIs

• Determination of Site Visit Issues

• Consensus Scorebook

• Category Champion leads Consensus discussion and scoring

• Feedback Report

• Verify / Clarify

• Training

• Managing for Innovation• Management by Fact• Societal Responsibility

• Ethics and Transparency• Delivering Value and Results

• Review Criteria

• Read Application

• Attend Training

• Training• Scoring Guidelines • Scorebook Navigator• MBNQA Website

Six Categories – ADLI; One Category - LeTCI

APPLICATION LEVELSAPPLICATION LEVELS

RESOURCES

SCORING FACTORS Process

Results

ENGAGEMENT COMMITMENT PROGRESS AWARD

• Org. Profile

• 10 Page App.

• No Site Visit

• Feedback Report

• Criteria • Glossary• Training• Scorebook

Navigator• Application

SCORES

PERFORMANCEEXCELLENCE

STRENGTHS

OPPORTUNITIES FOR IMPROVEMENT

ADLI (Basic)

LeTCI

• Org. Profile

• 20 Page App.

• No Site Visit

• Feedback Report

• Org. Profile

• 30 Page App.

• Site Visit (if Purchased by Applicant)

• Feedback Report

• Org. Profile

• 50 Page App.

• Site Visit

• Feedback Report

ADLI (Overall)

LeTCI

ADLI (some multiple)

LeTCI

Approach DeploymentLearningIntegration

LevelsTrendsComparisonsIntegration

APPLICATIONSRECEIVED October-March (5 cycles) OCTOBER

New Beginner Level (OP only)

Examiners’ Roles and Responsibilities

• Provide analysis and feedback to lead the organization to the next level of maturity

• Examination Process Team Members– Examiners– Senior Examiners– Scorebook Editor– Back-up Team Leader – Team Leader– Process Coach

• Each team has a Subject Matter

Expert or two (Can be any role)

Benefits of Being an Examiner

• Learn Quality Texas/Baldrige Performance Excellence Criteria; training and experience

• Learn validated best practices; leading organizations; all industry sectors

• Expand professional network; improve YOUR brand; valuable professional credential

• Develop assessment, analysis, writing, teamwork and leadership skills

• Give back to the community, state, and nation by helping organizations be successful through useful feedback (patriotism)

Conditions of Conditions of InvolvementInvolvement

Absolute Essentials to the Credibility, Absolute Essentials to the Credibility, Success and Prestige of the Assessment and Success and Prestige of the Assessment and

Feedback ProcessFeedback Process• Commit to the entire process (until the feedback is presented to the applicant)

• Maintain Confidentiality

• Complete the Training and Case Study

• Honor Time Commitments

• Represent Quality Texas (not your organization)

• Follow Code of Conduct; Avoid Conflicts of

Interest

Value for the Value for the Examiner/Sponsor Examiner/Sponsor

• Work with a diverse team; reach consensus

• Network with other quality/business experts to build professional friendships

• Understand/apply the Baldrige Framework to a variety of organizations; your own?

• Develop analytical/consensus-building skills

• Attend annual Texas Award Banquet and Conference (discounted rate) – June, 2016, Holiday Inn Riverwalk, San Antonio, TX

• Special recognition at conference

Examiner Career PathExaminer Position

Time Experiences

Examiner 1-3 years BU TL, FBW, 1 SV, Examiner Training Each Year

Senior Examiner

4-6 years TL, 2 SV+, Examiner Training Each Year

Alumni Examiner

6+ TL, Process Coach, 3+ SV, Examiner Training Each Year

Various Committees

Judge 6+ TL, Process Coach, 3+ SV, Examiner Training Each Year

Various Committees

Board of Directors

6+ Corporate Sponsorship, TL, Process Coach, 3+ SV, Examiner Training Each Year Various

Committees

Fellow 10+ Dedication and Continual Volunteerism for QTF Issues

QTF EXAMINATION& FEEDBACK PROCESS

INDEPENDENT REVIEW

QTF EXAMINATION&

FEEDBACK PROCESS

CONSENSUS REVIEW

QTF EXAMINATION& FEEDBACK PROCESS

Site Visit

Walk Through Baldrige Framework Booklet

Take a look at your Baldrige Framework Book

Baldrige Excellence Framework

• Criteria: Systems Perspective – 1

• Basic, Overall, Areas to Address – 2

• Point Values per Category/Item – 3

• Organizational Profile – 4-6• Categories 1-7 – 7-29• Scoring System – 30-33

Baldrige Excellence Framework

• Process Scoring Guidelines – 34• Results Scoring Guidelines - 35• Responding to the Criteria – 36-38

• Core Values and Concepts – 39-43• Changes from 2013-2014 – 44-46• Glossary of Key Terms – 47-54

QTF LEVELS

Award Levels 50 plus 5 (OP)

Progress Level 30 plus 5 (OP)

Commitment Level 20 plus 5 (OP)

Engagement Level 10 plus 5 (OP)

Beginning Level 5 (OP)

Award Next 50 + 5 (OP)

How Do I Evaluate Process Items?

Process items are evaluated using four factors:

•Approach

•Deployment

•Learning

•Integration

Approach (A)“Approach” comprises

• Methods used to carry out Process

• Appropriateness of methods vs requirements

• Effectiveness of the methods

• The degree to which the approach is repeatable and based on reliable data and information (i.e., systematic)

Deployment (D)

“Deployment” is the extent to which •Your approach addresses item requirements relevant and important to the organization

•Your approach is applied consistently

•The approach is used by all appropriate work units

Learning (L)Learning comprises•The refinement in approach through cycles of evaluation and improvement•The encouragement of breakthrough change through innovation, and,•The sharing of refinements and innovation with other work units and processes in the organization

Integration (I)“Integration” is the extent which

•Your approach is aligned with organizational needs (OP/ processes)

•Your measures, information, and improvement systems are complementary across processes and work units

•Your plans, processes, results, analyses, learning, and actions are “harmonized” across processes and work units to support organization-wide goals

Results: 45% of the Total Score

Results: 45% of the Total Score

LevelsLevels – Current performance on a meaningful measurement scale

TrendsTrends – Numerical data that show the direction and rate of improvements (slope over time)

ComparisonsComparisons – Your performance relative to that of other appropriate organizations, competitors or organizations similar to yours; relative to industry leaders or benchmarks

IntegrationIntegration – Measures (segmentation) addressing important performance requirements relating to customers, products/services, markets, processes, or action plans identified in your OP and in process items; future performance; harmonization across processes and work units to support organization-wide goals

Results Evaluation Factors LeTCI

Segmentation SegmentsSegments can be defined by, among other things:

Customers (students/families for Education; patients/families for HC)

Market or Product offerings By Location Workforce group (employees,

tenure, admin, hourly, etc.) Size of the group in

question

The applicant defines their segments (OP); we assess the results of those segments

Comparisons

ComparisonsComparisons can be defined by, among other things:

•Inside the industry•Competitive comparisons•Outside the industry

QTF/Baldrige Examination Process

TRUST THE PROCESS!(KEY TEACHING POINT)

Step 1: Read the Criteria.

Read Baldrige Framework for Award Level

QTF Beginner,Engagement, Commitment, or Progress Level Criteria

Step 2: Determine Most Relevant Key Factors

Four to Six Key Factors taken from OP, Eligibility, or from Application

Step 3: Read & Analyze the Application

Read the Application

Mark as appropriate

Step 4: Identify Strengths/OFIs

Around 6 commentsStrengths and OFIs

Step 5: Write Feedback Ready

Comments Remember: NERD!

N – nugget of importance

E – evidence/example to support comments

R – relevance to the applicant

D – Done!

Step 6: Determine the Scoring Range and

ScoreBest Fit

Don’t Block a Winner

Tie goes to applicant

Scoring: Not like your previous education!

Go to Process Scoring Page 34

UNDERSTANDING SYSTEMATIC APPROACHUNDERSTANDING SYSTEMATIC APPROACH

UNDERSTANDING DEPLOYMENTUNDERSTANDING DEPLOYMENT

UNDERSTANDING IMPROVEMENT/LEARNING/INNOVATIONUNDERSTANDING IMPROVEMENT/LEARNING/INNOVATION

UNDERSTANDING INTEGRATIONUNDERSTANDING INTEGRATION

Go to Results ScoringPage 35

What is different about Results Scoring?

Questions to this Point??

Examination

So let’s begin our Examination/Evaluation

Importance of the Organizational Profile

• Reader’s digest version of the application

• Sets the stage for what the applicant says is important

• Frames our comments (feedback report), strengths/OFIs, scoring, and Key Themes

• Assists with our scoring band descriptors (Award)

• Do not read into or take away from what is stated

How to determine Key Factors

• Review what the applicant says is important in the eligibility form

• Review the OP• Be aware throughout the application of other key factors that are discussed

• What is really important?• Do not rewrite the OP

Exercise: Determine Key FactorsActivity Time (min)

1. Select a scribe, timekeeper and reporter (1)

2. Discuss at your table as a group your thoughts on the key factors for this applicant from the Organizational Profile

10

3. Highlight Key Factors from the OP and select a few to write down; Table Anchors guide this process

4. Record the requirements on chart paper

20

10

5. Report Out (1 minute per table) 5

45 Minutes Total

45 Minutes Total

Step 1: Read the Criteria

Exercise: Step 1 Read the CriteriaActivity Time (min)

1. Select a scribe, timekeeper and reporter

(1)

2. Discuss at your table, (group) your thoughts on the key requirements for this Item 1.1 from Baldrige Framework Manual

3. Review individually your assigned Criteria Item [1.1a(1), (2), (3), 1.1b(1), (2)]

4. Record the requirements on chart paper

5. Report Out (1 minute per table)

30

10

10

10

60 Minutes Total

60 Minutes Total

Step 2: Determine Most Relevant Key Factors

Exercise: Step 2 Determine the Most

Relevant Key Factors Activity Time (min)

1. Select a scribe, timekeeper and reporter (1)

2. Review individually your assigned Criteria Item [1.1a(1), (2), (3), 1.1b(1), (2)]

5

3. Discuss at your table as a group and come to agreement on the relevant 4 to 6 key factors for your assigned item.

4. Write your selected key factors on a flip chart for use in subsequent exercises (capture key words and phrases)

20

10

5. Report Out (1 minute per table) 1045 Minutes

Total 45 Minutes

Total

Step 3: Read and

Analyze the Application

Exercise: Step 3 Read and Analyze the

Application Activity Time (min)

1. Select a scribe, timekeeper and reporter (1)

2. Review individually your assigned Criteria Item (1.1a(1), (2), (3), 1.1b(1), (2)

20

3. Discuss at your table as a group your observations on the applicant’s response against the Criteria questions.

20

4. Report Out (1 minute per table?) 5

45 Minutes Total

45 Minutes Total

Step 4: Identify Strengths/OFIs

Exercise: Step 4 Identify

Strengths/OFIsActivity Time

(min)

1. Select a scribe, timekeeper and reporter (1)

2. Use a round robin approach to discuss/highlight all potential strengths/OFIs identified by your table mates [1.1a(1) (2)(3), 1.1b(1)(2)]

15

3. From these, discuss at your table and select as a group, a total of around 2 strengths and OFIs that you feel are most important for the applicant

4. Record these on chart paper

15

5

5. Report Out (1 minute per table) 1045 Minutes Total

45 Minutes Total

Feedback Comments

Our KEY Product for the Applicant!

What Feedback Ready Comments Should

Do:Let the applicant know what it does well—and what it needs to improve to take it to the next level

1. Criteria requirements equitable assessment

2. The particular organization meaningful feedback

3. Assessment factors (ADLI)(LeTCI)

insights onorganizational maturity

Six Feedback Comment “Musts”

1. Polite, professional, non-prescriptive

2. Based on the Criteria, Key Factors, Scoring Guidelines

3. Addresses one topic per comment4. Begins with “nugget” that shows why

this is important to the applicant5. Has evidence/example and relevance

(actionable information) (based on scoring guidelines, Key Factors, Core Values, criteria notes)

6. Comments align with and support the score

Well-Written Comments: “NERD”• NN – Nugget – Up front comment based on the Item

Criteria or a Key Factor (i.e., explains why this is important to the applicant)

• EE – Evidence or Example - State what the applicant has (Strength) or is missing or not addressed (OFI)

• R R – Relevance – Why is this comment important? If an OFI, include actionable (but not prescriptive) information that will help the applicant to improve.

• DD – Done – Read to see if comment makes sense and helps the organization understand its current status

Comment WritingComment Writing

Elements of a well written comment

The Feedback Report: The ProductThe report should be:

•UsefulUseful to the applicant– Comments should be clear and actionable– Focused on the applicant’s most important key factors

•EncouragingEncouraging– Not adversarial (it’s not an audit)– Help applicant reach the next level– Make applicant an advocate for the program

•RespectfulRespectful– Polite in tone– Not judgmental or prescriptive

How to Give Useful FeedbackDo:

• Base your comments on the Criteria

• Reference the evaluation factors

• Include an opening “nugget” to give the comment significance

• Keep each comment to a single issue

• Make Key Factor references• Be accurate; check your statements

• Recognize the page limitations• Give benefit of the doubt• Be polite in tone• Include a few examples• Check spelling and grammar

Don’t:• Be judgmental or prescriptive

• Forget linkages • Forget the Key Factors• Stray from the Criteria• Make conflicting strength and OFI statements

• Forget to:– Check your score against the balance and content of comments

– Check your facts

– Check grammar and spelling

• Forget you’re providing a service

Sample Process Strength

2.2a(4) To mitigate the strategic challenge of competitors wishing to hire its engaged workforce, the applicant ensures that workforce plans support any needed changes. For example, through data and budget analysis and surveys, the applicant takes a proactive approach to workforce capacity and allocates instructional staff to areas of greatest need through “vertical teamwork.”

Nugget

Examples

Relevance

Sample Process OFI

4.2a(1)It is not clear how the applicant systematically transfers knowledge specific to the needs of parents and volunteers in support of the PhilP that all are accountable for student performance. For example, parents and volunteers do not appear to be included in teachers’ grade-level discussions, and parents do not appear to have access to teachers’ online forums, blogs, and classroom support server (Figure 4.2-1).

Example

NuggetRelevance

Sample Process Strength

2.2a(4) To mitigate the strategic challenge of competitors wishing to hire its engaged workforce, the applicant ensures that workforce plans support any needed changes. For example, through data and budget analysis and surveys, the applicant takes a proactive approach to workforce capacity and allocates instructional staff to areas of greatest need through “vertical teamwork.”

Relevance Nugget

Examples53 words, 372 characters/spaces

Sample Process OFI

4.2a(1)It is not clear how the applicant systematically transfers knowledge specific to the needs of parents and volunteers in support of the PhilP that all are accountable for student performance. For example, parents and volunteers do not appear to be included in teachers’ grade-level discussions, and parents do not appear to have access to teachers’ online forums, blogs, and classroom support server (Figure 4.2-1).

Relevance

Nugget

Examples

64 words, 415 characters/spaces

Sample Results Strength7.3a(1)Strong results for key measures of workforce capability and capacity may help strengthen the engagement factor of having sufficient resources to get the job done. Rates of certification (Figure 7.3-1) and student-teacher ratio (Figure 7.3-2) have improved over the periods shown, reaching 100% or close to 100% for all segments, and the student-teacher ratio in elementary and middle schools compares favorably to that of a Baldrige Award winner.

RelevanceNugget

Examples

Sample Results OFI

7.2aResults related to key student requirements, such as stimulating creative thought, treating students fairly, and maintaining a safe school, are missing. Tracking such results may give the applicant insights into how to retain families and how to attract families to the district.

Relevance

NuggetExamples

Step 5: Write Feedback Ready Comments

Exercise: Step 5 Write Feedback Ready Comments

Activity Time (min)

1. Select a scribe, timekeeper and reporter (1)

2. Draft feedback ready strength and OFI (2 groups)

3. Give feedback to table mates4. Redraft the comment, using their input5. Record the final feedback ready comment

on chart paper

15101010

6. Report Out (1 minute per table) 10

55 Minutes Total

55 Minutes Total

Introduction to Scoring• Process Scoring Guidelines and Results

Scoring Guidelines• Use Glossary for Definitions always• Process Scores Address: Approach, Deployment, Learning, Integration (ADLI)(ADLI)

• Results Scores Address: Levels, Trends, Comparisons, Integration (LeTCI)(LeTCI)

8.5 8.56.5

8.09.0

Step 6: Determine the Scoring Range and

Score

Look at Scoring Guidelines in Baldrige

FrameworkPages 34 Processand 35 Results

Exercise: Step 6 Scoring

Activity Time (min)

1. Select a scribe, timekeeper and reporter

(1)

2. Using your comments from Step 4 and the Scoring Guidelines, discuss as a group the most appropriate scoring range for your assigned Item

10

3. Next, discuss the appropriate score within that range

4. Record the scoring range and score on chart paper and be prepared to discuss why you chose that score

10

10

5. Report Out (1 minute per table) 535 Minutes Total

35 Minutes Total

Follow Six Step Process

• Complete for 2.1 [(2.1a(1)(2)(3)(4) and 2.1b(1)(2)]

• Complete for 5.2 [(5.2a(1)(2)(3)(4) and 5.2b(1)(2)(3)]

• Complete for 7.1 [(7.1a, b(1)(2), c]

• Complete for 7.5 [(7.5a(1)(2)]

Round-Robin Feedback

• What did you learn?

• Do you have confidence to begin/continue the assessment stages?

• Any final questions or concerns?

Blank

88

Scorebook Navigator Links to the IR and consensus

manuals for Scorebook

Navigator are on the Examiner

Resources page

Scorebook Navigator

Scorebook Navigator

Last Things…Almost!• Be sure to complete your training surveys and

forms!

• Please contact Quality Texas if you need help.

• Let’s thank our Training Faculty!

• Please take a few moments to clean up your areas

• Please remember to post your Plus/Delta comments!

• Have a safe trip home!

CONGRATULATIONS!

94

We appreciate We appreciate YOUYOU!!

Welcome to the Quality Welcome to the Quality Texas FamilyTexas Family