NLC Instructor: Instructional Design -...

Effective Feedback for Performance Evaluation 1 Rutherford, Naylor, & Wheeler

NLC Instructor: Instructional Design

Kara Rutherford, Beverly Naylor, Tammy Wheeler


TableofContentsProject Description .................................................................................................................... 4

Project title ................................................................................................................................ 4Information about the organization ........................................................................................... 4Information about the site client ................................................................................................ 5Information about the project .................................................................................................... 5Team members ......................................................................................................................... 7Project schedule ....................................................................................................................... 7

Performance Analysis ................................................................................................................ 9Gap Analysis ............................................................................................................................. 9Organizational Analysis .......................................................................................................... 10Summary: State the Case for Training ................................................................................... 18

Task Analysis ........................................................................................................................... 20Data Sources .......................................................................................................................... 20Completed Task Analysis ....................................................................................................... 22(3-5 things) of What Is Critical, Difficult, and Complex ........................................................... 34

Learner Analysis ...................................................................................................................... 35Brief Description of the Learners ......................................................................................... 35Sources and Methods ......................................................................................................... 35Data-Gathering Instruments ................................................................................................ 36

Findings and Implications ....................................................................................................... 37Relevant Skills ..................................................................................................................... 37Relevant Contextual Factors ............................................................................................... 40

Job-Focused objectives .......................................................................................................... 44Critical Tasks that the Training Will Address .......................................................................... 44Job-Focused Instructional Objectives ..................................................................................... 45

Performance Assessment Instrument .................................................................................... 47Objectives ............................................................................................................................... 47Type of performance assessment and rationale ..................................................................... 48Assessment instrument .......................................................................................................... 48Mastery level ........................................................................................................................... 49Job Aids .................................................................................................................................. 49Performance Assessment ....................................................................................................... 49

SET UP ................................................................................................................................. 49


INSTRUCTIONS AND SCENARIOS ............................................................................................ 50ASSESSMENT CHECKLIST ...................................................................................................... 52

Instructional Plan Worksheet .................................................................................................. 55Team Project Status Review # 1 ............................................................................................. 64

Overall Project Status in Relation to Schedule .................................................................. 64Actual or Potential Roadblocks ........................................................................................... 64Team Work Successes ......................................................................................................... 65Team Work Improvements ................................................................................................... 65

NLC Instructor Team Project Status Review # 2 ................................................................... 66Overall Project Status in Relation to Schedule .................................................................. 66Actual or Potential Roadblocks ........................................................................................... 66Team Work Successes ......................................................................................................... 66Team Work Improvements ................................................................................................... 67

NLC Instructor Team Project Status Review # 3 ................................................................... 68Overall Project Status in Relation to Schedule .................................................................. 68Actual or Potential Roadblocks ........................................................................................... 68Team Work Successes ......................................................................................................... 69Team Work Improvements ................................................................................................... 69

Appendix A: Interview Notes ................................................................................................... 71Appendix B: Chevalier’s Updated Behavioral Engineering Model (BEM) Order Completed for NLC Training Specialist ..................................................................................................... 77Appendix C: Survey Questions/Results ................................................................................. 80Appendix D: Extant Data Examples ........................................................................................ 91Appendix E: Job Aids .............................................................................................................. 92References .............................................................................................................................. 101


Project Description

Project title Effective Feedback for Performance Evaluations and Beyond

Information about the organization Northwest Lineman College (NLC) is an Idaho based for-profit vocational college offering training and education services to the power delivery industry - mainly training pre-apprentice to journey-level lineworkers. NLC currently has four campus locations: Meridian, Idaho (corporate headquarters); Oroville, California; Denton, Texas; Edgewater, Florida. NLC’s mission/vision and core values are important in providing insight into the organization: NLC’s mission/vision Northwest Lineman College will be a dynamic, visionary, and leading international educational institution providing the benchmark standard of safety, training, and products that exceed the expectations of customers in the power delivery and related industries. NLC’s core values Northwest Lineman College has an outstanding reputation throughout the power delivery industry, thanks to the high standards of its company, staff, and students. The Core Values and Mission Statement are used as the guiding principles by which NLC conducts training, business, and all endeavors. P.I.E. = PASSION, INTEGRITY, EXCELLENCE* PASSION Love what you do with PERSISTENCE, MOTIVATION, and the ATTITUDE to succeed. INTEGRITY Do the right thing—even when no one is watching—so TRUST, HONOR, and DEPENDABILITY are your hallmarks. EXCELLENCE Innovate always, leveraging ORIGINALITY and TEAMWORK, so your LEGACY of positive contribution to the industry is guaranteed. *(PIE is a mnemonic for the power formula, P (watts) = I (current) X E (voltage), which is a well-known and used formula in the power delivery industry): NLC hires qualified journey-level lineworkers as Training Specialists and is required by internal policy and state/accreditation agencies to train and certify Training Specialists to teach and train in all of its programs. An internal training department, Grid University, is responsible for designing and implementing all training for the Training Specialists. Currently, new-hire Training Specialists complete a 40-hour training program introducing education and training principles and practices that will get them started teaching in the


classroom and field. After the initial certification training, all Training Specialists are required to complete 40 hours of continuing education a year that builds on the introductory content of the initial certification.

Information about the site client The site client is the Chief Education Officer (CEdO) for the organization and holds a PhD in vocational education and has 37+ years of experience in higher education, both in teaching and administration. His major responsibilities are to oversee and supervise Grid University operations - specifically the instructional design process for faculty and staff training courses and programs - as well as review and advise on all externally offered program curriculum and training/education services. Client Stakeholders* Training Leads Campus Training

Specialist Campus

Lonnie M. Texas Charlie C. Idaho

Nick J. Idaho Mark G. Idaho

Rick C. Florida Darrow L. Idaho

Travis H. California Joe M. California

Jim L. California *The client stakeholders are listed for informational purposes only. The ID team is not responsible for organizing stakeholders.

Information about the project Request Background & Summary The initial request came to the Educational Leadership Committee (ELC membership consists of Training Leads for each campus, Director of Grid University, CEdO, and the VP of Curriculum & Instruction) directly from the President of the company. The NLC President based his request on feedback from several large industry employers who questioned the relevance of the behavior assessment course in the Electrical Lineworker Program (ELP). Specifically, their major concerns are: graduates with high behavior marks aren’t passing behavior-based interviews; graduates with high behavior marks are failing in apprenticeships. The behavior assessment grades are based on results from formal performance evaluations conducted throughout the term. There are several potential causes of this problem, some of which are:

• The test itself may be faulty. This would mean that the test isn’t an authentic measure of ELP skills: the test may not be valid. Addressing this issue is outside this team’s scope of work.

• Instructors may not be administering the test consistently. It could be that some instructors aren’t administering the entire test. Or, that different parts of the test are being administered in different ways. Or, that the checklists used to determine mastery are not clear. This may be within the scope of this project.


• Instructors may not be providing consistent feedback. This is within the scope of this project.

To summarize the request in the words of the President and the ELC (not direct quotes):

• Training Specialists need to provide feedback to students during evaluations in a way that:

o Promotes accountability o Is thoughtful and supportive o Doesn’t break a student down o Is truthful

Business Need The ELP program objective is to provide training and education that ensures graduates are highly qualified for initial employment, highly successful in completing formalized apprenticeship and training programs, and are prepared to pursue long, rewarding careers as lineworkers. This request is linked to the ELP program objective and to the overall mission/vision and core values of the company. The NLC President is concerned that the low quality of performance evaluations is harming the reputation of NLC and its graduates within the industry and is not meeting the program objective. The demand for NLC graduates in the industry is what drives enrollment in the ELP, which is a major revenue source; therefore, maintaining their reputation in the industry and exceeding customer expectations is of upmost importance. Performance Gap Summary

Target Population Performance Standard

Desired Performance

What NLC wants their Training Specialists

to do is provide effective feedback during formal student evaluations

that is: truthful feedback based on specific and noted observations of the student’s progress in the program; setting the expectation for student accountability for their actions/non-actions; empathetic yet firm on discussed plans for improvement if necessary.

Actual Performance

What NLC Training Specialists

are doing now is reading through evaluations and providing standardized feedback from the forms

that is: generic and not indicative of actual student performance.


Team members Team Member Email Phone Emergency Contact

Beverly Naylor [email protected] (H) 613-834-9927 (C) 705-817-5628

Dan Naylor (husband) 613-834-9927

Kara Rutherford [email protected] (C) 208-392-8097 Jonathan Cano (husband) 208-409-7456

Tammy Wheeler [email protected] (C) 804-564-4308 David Wheeler (husband) 804-874-9266

Team Contact Escalation Plan If a team member disappears from contact, Team NLC follows this contact escalation plan:

1. Send an email and copy other team members. If the team member in question does not respond within 24 hours, proceed to next step.

2. Send text message to cell phone. If the team member in question does not respond within 12 hours, proceed to next step.

3. Call team member and leave a message. If no answer, immediately contact other team members to formulate alternate plan which the team then expedites.

4. Email the team member in question to discuss the team’s decision to proceed with alternate plan.

5. If no response is made within 24 hours after emailing alternate plan, a team member calls the emergency contact.

Project schedule Team NLC Instructor recognizes that the dates listed for site client review are tentative dates that may need to be adjusted to comply with the client’s schedule. Team review Site client

review Resubmission Final course

deadline

Project description Thursday 8 September

N/A Thursday 15 September

Friday 9 September

Project status review Thursday 8 September

N/A N/A Friday 9 September

Team charter Thursday 8 September

N/A N/A Friday 9 September

Team meeting 1 with Steve Friday 8 September

N/A N/A Sunday 11 September

Performance analysis Sunday 25 September

Wednesday 12 October

Friday 7 October

Friday 30 September


Task analysis Tuesday 27 September


Friday 7 October

Friday 30 September

Project status review 2 Thursday 29 September

N/A Friday 7 October

Friday 30 September

Team meeting 2 Sunday 25 September

N/A N/A Sunday 2 October

Learner analysis Sunday 9 October


Friday 21 October

Saturday 15 October

Objectives Sunday 16 October


Friday 28 October

Saturday 22 October

Performance assessment Sunday 23 October


Friday 11 November

Saturday 29 October

Instructional planning worksheet

Tuesday 8 November

Wednesday 9 November

Friday 18 November

Friday 11 November

Project Status Review 3 Tuesday 8 November

N/A Friday 18 November

Friday 11 November

Team meeting 3 with Steve Sunday 6 November

N/A N/A Sunday 13 November

Instructional plan Sunday 20 November


Friday 9 December

Monday 28 November

Instructional materials (optional)

Sunday 20 November


Friday 9 December

Monday 28 November

Formative evaluation (Review of another team)

Sunday 4 December

N/A Friday 9 December

Monday 5 December


Performance Analysis Effective Feedback for Performance Evaluations & Beyond

Gap Analysis

Target Population Performance Standard

Desired Performance

What NLC wants their Training Specialists

to do is provide effective feedback during formal student evaluations

• Assesses cumulative performance level including grades, competencies, certifications, safety, attitude, housekeeping, excellence, and integrity with at least a moderate level of inter-rater reliability (based on Cohen’s interpretation of kappa, moderate level of agreement, k = 0.41 – 0.60);

• Completes the ELP performance rubrics based on assessment in a timely manner (completing forms in no more than 15 minutes per student – for 32 students);

• Conducts the face-to-face formal evaluation, in an empathic, firm, and consistent manner, using the completed ELP performance rubric effectively (completing each student meeting in 10 minutes or less; student’s report 80%


satisfaction with evaluation via post evaluation survey).

Actual Performance

What NLC Training Specialists are doing now is reading through evaluations and providing standardized feedback from the forms

that is generic and not indicative of actual student performance.

Organizational Analysis About NLC Northwest Lineman College (NLC) is an Idaho based for-profit vocational college offering training and education services to the power delivery industry - mainly training pre-apprentice to journey-level lineworkers. NLC currently has four campus locations: Meridian, Idaho (corporate headquarters); Oroville, California; Denton, Texas; Edgewater, Florida. NLC’s mission/vision is: Northwest Lineman College will be a dynamic, visionary, and leading international educational institution providing the benchmark standard of safety, training, and products that exceed the expectations of customers in the power delivery and related industries. NLC hires qualified journey-level lineworkers as Training Specialists and is required by internal policy and state/accreditation agencies to train and certify them to train in all of its programs. An internal training department, Grid University is responsible for designing, implementing, and assessing all training for the Training Specialists. Grid University assesses Training Specialists during their initial certification process to determine readiness to begin training in the educational programs and services. Currently, new-hire Training Specialists complete a 40-hour training program introducing education and training principles and practices that will get them started teaching in the classroom and field. After the initial certification training, all Training Specialists are required to complete 40 hours of continuing education a year that builds on the introductory content of the initial certification. Currently, Training Specialists complete a 25-minute lesson on the ELP performance rubric form and evaluation process as part of 1-2 hour session on the purpose of assessments and evaluations in education. This 25-minute lesson provides only a brief overview of the ELP performance rubric and is general information: it does not provide instruction on how the rubric is to be completed or the processes involved in completing the ELP performance rubric.


Information About the Site Client The NLC President has identified the site client for this project as the Chief Education Officer (CEdO) for the organization. The CEdO holds a PhD in vocational education and has 37+ years of experience in higher education, both in teaching and administration. His major responsibilities are to oversee and supervise Grid University operations - specifically the instructional design process for faculty and staff training courses and programs - as well as review and advise on all externally offered program curriculum and training/education services. Why Closing the Performance Gap is Important to NLC Based on informal feedback from industry employers of NLC graduates, the NLC President has identified (in an informal non-qualitative/non-quantitative analysis) that there is a current performance gap of instructors not providing consistent feedback to students, which affects NLC’s reputation as an educational leader in the electrical lineworker industry. NLC relies on this reputation to maintain enrollments in its educational programs and services as most incoming students report that they heard about NLC from an NLC graduate or someone in the industry. Additionally, NLC’s mission/vision statement indicates a clear commitment to customer satisfaction, indicating they want to “exceed the expectations of customers in the power delivery and related industries.” Therefore, closing this performance gap is an important step to increasing customer satisfaction in the marketplace. Measures Associated with Performance Currently, NLC does not collect specific data on the student performance evaluation process or how it measures its reputation perception. The NLC team will measure success by:

● Collecting and analyzing the performance forms to determine consistency using the Kappa statistic, which measures inter-rater reliability.

● Surveying students to analyze student satisfaction of the evaluation process. ● Observing Training Specialist preparing for and conducting the student performance evaluations.

Team NLC Instructor collected data from the sources listed below to analyze both the performance and causes of the stated performance gap.

Data sources Data gathering methods Findings

Phone interview BSU ID Team interviewed Frank S. (pseudonym), Training Specialist for NLC (based out of Texas campus),

Environment Information: Each performer felt improvement could be made in regards to providing feedback. The exemplary


identified as a newer Training Specialist

performer noted expectations are verbally communicated but not written out.

Resources: Both performers felt data was available. There was some discrepancy on how to go about finding the data. Processes and procedures have changed over time.

Incentives: Both performers indicated that no incentives, monetary or non-monetary are in place. Occasional verbal praise is given. The organization does not have any overt rewards or punishments for desired or undesired behavior; however, the performance form and process itself may be covertly punishing desired behavior because rating a student as unacceptable requires more time and effort to complete the forms and provide feedback.

Motives: Both performers felt verbal and written feedback is an essential part of the education they provide to future linemen.

Capacity: There are no issues in regard to capacity or ability to learn.

Knowledge/Skills: It’s difficult to truly know a student’s knowledge, skill, and behavior if you don’t interact with the student on all levels in which they are evaluated.

Phone interview BSU ID Team interviewed Sam M. (pseudonym), Training Specialist for NLC (based out of Idaho campus), identified as an exemplary performer

Survey BSU ID Team conducted 3 surveys (see Appendix C):

• Survey 1 was sent to all Training Specialists and had questions based on Chevalier’s Updated Behavioral Engineering Model (BEM) for NLC Training Specialist (Appendix C).

Survey 1 (survey sent to 24 Training Specialists, 18 responded – 75% response rate). Survey highlights/findings:

• The majority of Training Specialists have performed 4 or more performance evaluations (83%) and report they know what is expected of them when conducting performance evaluations (89%).

• There is somewhat of split on whether regular and meaningful feedback is given on their performance (28% disagree/strongly disagree that they get feedback while 39% agree/strongly agree they get


feedback).

• While Training Specialists report they are very comfortable providing feedback to students during performance evaluations (94%), the response to how they feel about providing written and verbal feedback during evaluations indicates an inconsistency (33% feel average or fair and 67% feel good or excellent).

• 61% of training specialist report they agree/strongly agree that they’ve received adequate training while 39% report they are neutral or disagree that they’ve received adequate training.

• Survey 2 was sent to all Campus Directors and Training Leads, who are responsible for their campus’ Training Specialists ability to perform as expected and had questions based on the Job Aid / EPSS Analysis (Appendix C).

Survey 2 (survey sent to 11 Campus Directors/Training Leads, 8 responses – 72.3% response rate). Survey highlights/findings:

• 75% of Campus Directors/Training leads report that it’s important to provide feedback to Training Specialists about their performance level conducting student performance evaluations.

• Campus Directors/Training leads report are evenly split on whether they think Training Specialists can provide feedback to students without a significant amount of practice (50% agree, 50% disagree).

• Campus Directors/Training Leads are split on whether a job aid could be used in place of formal training sessions (38% disagree while 62% agree/strongly agree).


• Survey 3 was sent to all Training Specialists to collect information for a learner analysis with more specific questions about completing the ELP performance rubric form.

Survey 3 (survey sent to 24 Training Specialists, 15 responded – 62.5% response rate). Survey highlights/findings:

• The majority of Training Specialists report that the behaviors criteria is the most difficult to accurately assess using the ELP performance rubric form (60%).

• The majority of Training Specialists report their performance level on accurately assessing student behavior as average or good (87%).

• 93% of Training Specialist report they are very open to receiving feedback on performance.

• 100% of Training Specialists report that training on assessing student performance is relevant and meaningful to them and their job.

• 67% of Training Specialists report there are barriers to applying what they learn to the workplace.

• 53% of Training Specialists report that it’s difficult to get enough information about unfamiliar students in order to accurately assess behavioral performance.

NLC Webpage Extant Data • NLC history

• NLC spirit

• NLC core values, mission

• Electrical Lineworker Program (ELP) objective

Electrical Lineworker Program (ELP) Performance Rubric and Coordinating Improvement Planners for Academics, Skills, and

Extant Data • Criteria (Knowledge - Grades, Skills - Competencies, Certifications, Behaviors - Safety, Attitude, Housekeeping, Excellence, Integrity)

• Performance Level (Unacceptable, Expected,


Behavior Exceptional)

• Required Probationary Action Steps for Academic, Skills, and Behavior Improvement

Updated Behavioral Engineering Model for Training Specialists Environment Information Resources Incentives

• 89% of Training Specialists surveyed agree they know the level of performance expected when conducting, and providing effective feedback during formal student evaluations.

• 39% of Training Specialists surveyed get regular and meaningful feedback on level of performance on providing feedback during formal performance evaluations.

• 88% of Campus Directors and Training Leads surveyed agree Training Specialists are expected to provide feedback to students during formal student performance evaluations quickly and smoothly.

• 63% of Campus Directors and Training Leads surveyed disagree that the environment or situation is unpredictable.

• 50% of Training Specialists surveyed agree that there are adequate resources (time, forms, help from others) available to help provide feedback to students during formal student performance evaluations.

• Electrical Lineworker Program (ELP) Performance Rubric and Coordinating Improvement Planners for Academics, Skills, and Behavior provide a guide for the Training Specialists during the student performance evaluations.

• There is no mentor program in place to provide coaching to new Training Specialists.

• 61% of Training Specialists surveyed agree there are non-monetary incentives for good or high levels of performance in providing feedback to students during formal student performance evaluations.


Individual Knowledge/Skills Capacity Motives

• 83% of Training Specialists surveyed have conducted 4 or more formal student performance evaluations.

• 50% of Training Specialists surveyed agree that the processes and procedures for conducting student evaluations have remained consistent over time.

• 60% of Training Specialists surveyed

indicate the behaviors criteria are the most difficult to assess.

• Training Specialists have the capacity or ability to learn.

• All Training Specialists are

previous linemen that possess knowledge of regulations that apply to electrical linework (OSHA, NESC, etc.). All possess a Journeyman lineman certificate of completed apprenticeship.

• Educational level breakdown:

o H.S. diploma/GED: 45.8%

o Some college or Vocational/Technical program: 37.5%

o Associate degree: 4.2% o Bachelor degree: 12.5%

• 66% of Training Specialists surveyed feel good or excellent about providing written and verbal feedback to students during formal student evaluations.

• 17/18 Training Specialists surveyed are comfortable with providing feedback to students that need improvement in the knowledge, skills, and behavior phase.


Job Aid / EPSS Analysis Question Yes No How do you know? What evidence (data/findings) can you provide

to support your answer? 1. Is it important for people to practice the

task to mastery? Or can people perform the job task without a significant amount of practice?

X The Training Specialists reported in the survey conducted that the task of conducting formal student evaluations involves numerous challenging tasks such as: working with a tight schedule, being organized and prepared, and getting other Training Specialists together to discuss student behaviors. Additionally, survey results from Campus Directors and Training Leads indicate a strong agreement (62.50%) of the importance in practicing providing feedback to students during formal evaluations to a level of mastery. 50% of Campus Director and Training Leads believe a significant amount of practice is needed to provide effective feedback to students.

2. Is it important for people to obtain both: ° Coaching (including error detection

and correction) as they perform a task?

° Delayed feedback (about the extent to which performance met standards and how to improve it) after they perform a task?

X The target audience for this training worked in the field as linemen prior to becoming Training Specialists. The typical performer noted he’s not as comfortable in the classroom. Learners will need to have multiple opportunities to see a variety of scenarios where a student needs improvement and coaching of how an exemplary performer guides the student to providing feedback needed for improvement based upon the performer’s observations and documentation. Opportunities should be provided to practice scenarios with coaching and feedback prior to working in the classroom to ensure mastery of providing feedback to linemen. The training will provide confidence and preparation necessary to respond to various student scenarios the Training Specialist faces during the evaluation process.

3. Is it important to provide learners with a safe, controlled environment in which they can make mistakes as they learn?

X The training environment provides the Training Specialist with a safe controlled environment where areas of weakness can be improved upon and provided feedback without affecting the linemen students. 80% of Training Specialists indicate that the learning environment is a safe and positive environment.

4. Is there an expectation of interactive learning?

X Results from Survey 2 indicate the campus/program leaders agree (75%) and strongly agree (25%) that there is an expectation of interactive learning when it comes to the student performance


evaluations. 5. Are people required to perform the task

quickly and smoothly in the workplace? X Performers mentioned they are given 10 minutes per student during the

formal evaluation process. 6. Are people required to perform the task

in situations that are unpredictable? X Training Specialists perform in a relatively predictable environment. As

such, these performers could benefit from the use of a job aid. 7. Does one or more of the following

reasons for NOT relying on a job aid alone exist in the on-the-job environment: ° A physical barrier that makes it

difficult to use a job aid on the job ° A social barrier that makes it

difficult to use a job aid on the job

X We believe the ELP Performance Rubric and coordinating planners are a form of a Job Aid already. Although we have identified possible Job Aids not in place today to help facilitate the process of delivering effective feedback. Topics for Job Aids include:

● Performance Levels (Unacceptable, Expected, Exceptional)

● Assessment Decisions Flowchart

● Evaluation Site Setup (Outside Team’s Scope of Work)

● Veteran Instructor Guide (Outside Team’s Scope of Work)

Summary: State the Case for Training Based on the Gap, Cause, and Organizational analysis results, Team NLC Instructor concludes that the performance gap can be closed with training and job aids.

• Currently, typical performers, especially novice Training Specialists, conduct student performance evaluations in a very generic and impersonal manner. There is a tendency to rate students as having met the expected performance level because it is easier to do in the amount of time provided and with the amount of information available. Exemplary performers conduct performance evaluations in a very personal and specific manner and tend to rate students more reliably using their time to focus on consulting with other Training Specialists to get information when required and accurately using the ELP performance rubric to rate students appropriately.

• The performance analysis indicates that experienced Training Specialists are adequately trained, however, there are indications that novice Training Specialists require more information, training, feedback, and time/collaboration to compile the evaluations, especially for students they don’t know.

• There is currently a brief introduction to the ELP performance rubric form and evaluation process; however, there are no opportunities to practice or experience the many varied situations that arise when preparing and conducting the performance evaluations.

• Training Specialists report that the behavior criteria assessments are difficult and complex to complete accurately. The task of assessing behavior requires complex decision making and situation recognition. A training program with a focus


on using job aids to help complete the behaviors criteria section of the ELP performance rubric as well as practicing complex decision making and situation recognition is necessary to assess behavior needed to close the gap.

Closing this performance gap is important to the organization because it’s directly linked to maintaining or increasing customer (graduates and industry employers) satisfaction. Graduates want a return of investment on their training. Industry employers want to be assured that the graduates are performing at a level that meets recognized industry standards.


Task Analysis Effective Feedback for Performance Evaluations and Beyond

Data Sources Selected Task

Analysis Method Rationale Data Sources

Procedural task analysis with if-then and complex tables

• This process consists of a series of sequential steps and decision-making processes;

• The task analysis uses if-then decision to represent simple decision-making processes;

• The task analysis uses complex tables to represent the questions that exemplary performers ask themselves to reach a decision.

• Interview with NLC exemplary instructor (Charlie,10+ years of experience as a Training Specialists at NLC);

• Interview with NLC novice instructor (George, little over 1-year experience as a Training Specialist at NLC);

• Survey of NLC Training Specialists;

• NLC documentation.

Task Characteristics

Task Characteristic Description Major tasks The major tasks involved in providing feedback

include:

• Reviewing the Student File Packet and verifying contents with other instructors to assess student performance;

• Completing the NLC Electrical Lineworker Program performance (ELP) rubric correctly and consistently;

• Preparing the interview site;

• Debriefing the student based on the completed ELP rubric.

Goals for each major task • Ensure the students’ performance is assessed consistently based on the data provided from the Student File Packet and from interviewing other instructors;

• Ensure the ELP performance rubric is completed correctly based on data provided/collected;

• Ensure the students receive objective feedback based on actual performance results.


Task Characteristic Description Cues • The formal student performance evaluations are

scheduled to occur on the Thursday of weeks 6, 10, and 15 of the ELP (program is 15 weeks total);

• Preparation and completion of forms is scheduled to occur the day prior to the scheduled formal evaluations.

Resources • The ELP performance rubric is very detailed and has evolved to be a sort of job aid in and of itself;

• The Student File Packet contains: o A blank ELP performance rubric with

student name labeled at top; o Student grades in all academic courses up

to the date of the evaluation; o Student progress in all competency skill

areas up to the date of the evaluation; o List of Safety Council members; o Student spot evaluations for:

§ Recognition of Exceptional performance

§ Sticker records § Probation (academic, skill, or

behavioral) § Not meeting written program

requirements § Absence counts § Tardy counts

• 8.5 hours of preparation time, scheduled to occur the day prior to giving the feedback;

• 8 hours of time scheduled to conduct the face-to-face performance evaluation meetings.

Standards that the completed task should meet • Consistently assess student’s cumulative

performance level including grades, competencies, certifications, safety, attitude, housekeeping, excellence, and integrity with at least a moderate level of inter-rater reliability (based on Cohen’s interpretation of kappa, moderate level of agreement, k = 0.41 – 0.60);

• Complete the ELP performance rubrics based on assessment in a timely manner (completing forms in no more than 15 minutes per student – for 32 students);

• Conduct the face-to-face formal evaluation, using the completed ELP performance rubric effectively (completing each student meeting in 10 minutes or less; student’s report 80% satisfaction with evaluation via post evaluation survey).


Task Characteristic Description Prerequisite skills • Be able to organize/sort and reference information

provided; • Be able to communicate clearly (enunciate,

smooth flow/not stumbling with words, interact with others, verbally, in writing, and through active listening.

Additional factors to consider related to Task Analysis: Consequences of not accurately completing the tasks

• Students are provided with misleading feedback.

• Students may believe that they are performing to a higher standard than what they are actually attaining.

• Weak students may feel isolated and alienated, feeling that they do not have support.

• Industry leaders hiring NLC are dissatisfied with the performance of NLC graduates.

Assumptions made by Team NLC Instructor • Instructors want to aid in student learning and development.

• An instructor may not have observed a specific student performing in the field or interacted with a specific student prior to the formal assessment.

• An instructor may have to depend on the Student File Packet and other instructor input for information to include in the ELP performance rubric.

Key to symbols

' Hot tip: Hints or advice about performing the task

F Note: Additional information

M Caution: Use for safety and potential mistakes

How to read this task analysis:

Yellow activities are performance activities that are critical to providing feedback.

Blue tasks are activities that do not require training and can easily be managed by the use of job aids or fall outside the scope of the team’s 2-3 hour training session.

Completed Task Analysis Conducting Formal Student Performance Evaluations 1.0 Prepare to assess student performance based on Student File Packet data.

1.1 Obtain the list of students assigned to you for evaluation and their corresponding Student File Packets.

1.2 Verify that the Student File Packet contains all required documents necessary to assess performance for each student on the assigned list.

F Note: The Student File Packet is provided by the academic assistant and contains:


• A blank ELP performance rubric form with student’s name labeled at the top • A list of Safety Council Members • Academic Records (current grade report) • Skill Records (competency and certification progress report) • Behavioral records (spot evaluations, probations, attendance records)

1.2.a If any documents are missing ask the academic assistant to provide the necessary information.

1.3 Review the list of assigned students to identify unfamiliar students (unfamiliar students are those with whom you’ve had little to no interaction with in the classroom, field, and/or behaviorally). 1.3.a Highlight unfamiliar student names on the list to indicate interviews with

other instructors will be necessary to complete the assessment and completion of the ELP performance rubric form.

1.3.b If there are no unfamiliar student names on the list, proceed to step 2.0 and follow directions for familiar students throughout the task analysis.

1.4 Conduct interviews about unfamiliar students.

2.0 Use the Student File Packet to complete the ELP performance rubric for each student on the assigned list (see Appendix XX (not included in this document) to view entire form).

F Note: If you have a list of 32 students, you will have just under 16 minutes to prepare and complete the ELP performance rubric form for each student on your list. If you have 16 students, you will have just over 30 minutes to prepare and complete the form for each student on your list. Don’t rush, but stay focused and remember to consult and collaborate with other instructors to get all the information you need to complete the forms accurately and consistently. The majority of instructors report that the full 8.5 hours is enough time to prepare and complete the forms.

2.1 Check the appropriate box in the Formal Evaluation section to identify the type of formal evaluation.

F Note: Week 6 evaluations are 1st of term and Week 10 evaluations are 2nd of term. Disregard the spot evaluation checkboxes for the formal evaluation.

Image for step 2.1

2.2 Complete the Knowledge Criteria Section

2.2.a Review the student’s current grade report to assess Knowledge criteria performance.

2.2.a.1 If the student name is highlighted as unfamiliar, consult with instructors who are familiar with the student.

2.2.a.1.i Obtain additional feedback on performance in the classroom.


2.2.a.1.ii Take written notes of any information that will be relevant to provide feedback for the student verbally during the formal evaluations.

2.2.b Based on academic course scores, circle the appropriate box on the ELP performance rubric form.

Image for Step 2.2.b

2.2.c If student is marked “Unacceptable” see step 2.6 to complete the ELP Academic Improvement Planner prior to moving on to Step 2.3.

2.3 Complete the Skill Criteria Section

2.3.a Review the student’s current competency and certification progress report to assess Skill criteria performance.

2.3.a.1 If the student name is highlighted as unfamiliar, consult with instructors who are familiar with the student.

2.3.a.1.i Obtain additional feedback on performance in the field.

2.3.a.1.ii Take written notes of any information that will be relevant to provide feedback for the student verbally during the formal evaluations.

2.3.b Based on competency and certification scores, circle the appropriate box on the ELP performance rubric form.

F Note: In week 6, students should be 90% complete with level 1 competencies/certifications. In week 10, students should be 100% complete with level 1 competencies/certifications and 80% complete level 2 competencies/certifications.

Image for Step 2.3.b

2.3.c If a student is marked Unacceptable, use decision table to proceed:

If the student was Then

also marked Unacceptable in the Academic criteria section of the ELP performance rubric form

Don’t proceed completing the Skills Improvement Planner – instead:

Bring the Student File Packet and ELP performance rubric to Training Lead (the


Training Lead is responsible for completing the rest of the form and conducting the evaluation – the student may be terminated per the satisfactory progress policy of the program).

not marked Unacceptable in the Academic criteria section of the ELP performance rubric form

Proceed to step 2.7: Complete the ELP Skills Improvement Planner prior to proceeding to Step 2.4.

2.4. Complete the Behaviors Criteria Section 2.4.a Use the following decision table to assess behaviors criteria on the ELP

performance rubric:

Precursor Action Results Interpretation Exceptional Behavior

2.4.a.1 • Student has one or

more spot evaluations for recognition of Exceptional performance/stickers.

F Hot tip: Reference Appendix XX (not included in this document) for sticker descriptions, taken from the program orientation guide provided to student during first week of class).

M Caution: If a student has one or more Exceptional spot evals, but also has one or more Unacceptable spot evals, bring the student information and the evaluation to the Training Lead to assist with the rest of the assessment. This situation is very rare and requires additional scrutiny by

• If spot eval form(s) was

completed by another instructor, review any notes provided and consult with instructor who completed the form to obtain any additional facts, etc.;

• If spot eval form(s) was completed by you, think back to the circumstance that led you to perform the spot evaluation;

• Read the Expected and Exceptional rubric boxes carefully to determine the appropriate rubric box to mark – depending on which category the spot evaluation is for, some categories may still be marked as Expected;

• Identify which behaviors category the spot evolution/or own assessment fits into:

Safety examples • Serves as a member of

safety committee; • Consistently observed

pointing out safety issues even when they don’t know they are

• Instructor

accurately and consistently assesses student behavior when spot evaluations are present for Exceptional behavior;

• The correct behavior category is identified consistently amongst students being evaluated.

• One or more

spot evaluations for recognition of Exceptional performance, indicates the student is performing at the Exceptional level for one or more of the given categories;

• Reviewing the spot eval forms with the instructor who completed them allows the evaluating instructor to become more familiar with the student’s behaviors and skilled in giving consistent and accurate


leadership to avoid mistakes in the process.

being observed; • Consistently observed

watching out for his/her crew and others – attentive and engaged in the field at all times.

Attitude examples • Has one or more

passion sticker spot evals;

• Consistently positive and upbeat – encourages his/her team and others;

• Takes on role of peer teaching, coaching without prompting;

• Takes on challenges positively and rallies his/her crew to succeed;

Housekeeping examples • Cleans and/or picks up

shed/tool room/field area/classroom without prompting;

• Checking equipment, tools, and materials to ensure they are put away and not damaged without prompting;

Excellence Examples • Has one or more

excellence and/or knowledge and/or skills sticker spot evals;

• Consistently engaged in classroom, field, lab environments – answers questions when asked, asks appropriate questions, leads group discussions or activities;

• Has aced all academic course tests in one or more subjects – verify on grade report.

Integrity Examples • Has one or more

integrity sticker spot evals;

• Has been observed

feedback.


turning in lost tools, making sure sheds are locked.

• Circle the correct rubric

box based on the overall assessment for each Behavior criteria category.

Areas for Improvement 2.4.a.2 • Student has one or

more spot evaluations for probations, not meeting written program requirements, excessive absences or tardies.

• If the spot eval form was

completed by another instructor, review any notes provided and consult with instructor who completed the form to:

• Obtain any additional facts, etc.

o Determine if the student met the requirements of any previous spot evaluations

o OR if the spot eval was completed by you, think back to the circumstance that led you to perform the spot evaluation;

• Read the Unacceptable and Expected rubric boxes carefully to determine the appropriate rubric box to mark – depending on which category the spot evaluation is for, some categories may still be marked as Expected.

M Caution: It’s very rare that a student with an Unacceptable spot eval will be assessed as Exceptional in any other Behavior criteria. Consult with the Training Lead to avoid making

• Instructor

accurately and consistently assesses student behavior when spot evaluations are present for behavior that requires improvement;

• The correct behavior category is identified consistently amongst students being evaluated;

• If any category is marked Unacceptable, the behavior improvement planner is completed to offer the student additional feedback and options for improvement – see step

• One or more

spot evaluations for probation, not meeting requirements, etc. indicates the student is performing at the Unacceptable level for one or more of the given categories; however,

• If progress is shown in follow-up documentation and the student is following improvement guidelines, the student is performing at the Expected level for one or more of the given categories;

• Reviewing the spot eval forms with the instructor who completed them allows


mistakes in your assessment if this occurs.

• Identify which behaviors category the spot evolution/or own assessment fits into:

Safety examples • Not wearing gaff guards,

climbing with proper gloves, or wearing the required personal protective equipment (PPE);

• Injury occurred after coaching/feedback from instructor, e.g. cut to hand/thigh in underground, large splinters/injuries due to gaff outs on pole;

• Not completing tailboard per NLC standards/requirements.

Attitude examples • Negative or short

tempered with his/her crew or others;

• Complains regularly and/or blames others;

• Reluctant to do the what’s expected in class, field or lab – answering questions with “I don’t know” saying “I don’t want to”, etc.;

• Often uses foul language after being warned;

Housekeeping examples • Misplaces name tag or

doesn’t place on tables in classrooms without prompting;

• Crew or others are regularly (2 times or more) picking up after the student because student skips out, e.g. tools, equipment, etc.

2.8. the evaluating instructor to become more familiar with the student’s behaviors and skilled in giving consistent and accurate feedback.


Excellence Examples • Not ready to perform

competencies or certifications during verification/check-off weeks;

• Excessive absences or tardies (3 or more absences, 5 or more tardies).

Integrity Examples • Does not participate in

community service projects (indicated in grade report);

• Caught lying, cheating, or stealing;

• Continues to use foul language after being warned;

M Caution: It’s rare that a student with a spot evaluation for an integrity violation of cheating or stealing will still be enrolled in the program by the time the formal evaluation occurs. Refer to the student conduct policy when assessing this Behavior criteria category (Appendix XX -not included in this document). Consult with the Training Lead to avoid making mistakes in your assessment if this occurs.

• Circle the correct rubric box based on the overall assessment for each Behavior criteria category.

2.4.a.3 • Student has no spot

evaluations for Exceptional behavior or behavior that

• If student is unfamiliar to

you, consult with other instructors to get more

• Instructor

accurately and

• The lack of

documentation in the


needs improvement. information, feedback, etc.;

• If the student is familiar to you, try to recall specific interactions with the student to note for verbal feedback;

• Read the Unacceptable and Expected rubric boxes carefully to assess the student’s behavior accurately and consistently

F Note: It’s rare that a student is performing at the Unacceptable level if there are no previous spot evaluations or probations in the Student File Packet.

• Identify which behaviors category the spot evolution fits into (safety, attitude, housekeeping, excellence, integrity);

• Circle the correct rubric box based on the overall assessment.

consistently assesses student behavior when no documentation is present in the student file;

• The correct behavior category is identified consistently amongst students being evaluated;

• If any category is marked Unacceptable, the behavior improvement planner is completed to offer the student additional feedback and options for improvement – see step 2.8.

student file, indicates the student is performing in the Expected rating for one or more of the given categories;

• Reviewing the student’s performance level with instructors who are familiar with the student allows the evaluating instructor to become more familiar with the student’s behaviors and skilled in giving consistent and accurate feedback.


Image for step/table 2.4.a

2.4.b If a student is marked Unacceptable use decision table to proceed:

If Then

The student was also marked Unacceptable in the Academic OR the Skill criteria section of the ELP performance rubric form

Don’t proceed completing the Skills Improvement Planner – instead:

M Bring the Student File Packet and ELP performance rubric to Training Lead (the Training Lead is responsible for completing the rest of the form and conducting the evaluation – the student may be terminated per the satisfactory progress policy of the program).

If the student was not marked Unacceptable in the Academic OR Skill criteria section of the ELP performance rubric form

Proceed to step 2.8: Complete the ELP Behavior Improvement Planner prior to proceeding to Step 2.5

2.5 Complete the accomplishments section 2.5.a Check the box if the student is the Student Body President or Vice

President (indicated on grade report) and/or check the Others box if the student is on the Safety Committee – write Safety Committee on the indicated line.

2.5.b Check boxes for every spot evaluation for stickers present in the Student File Packet and tally results on the indicated line.

2.5.c Complete the Others box to specify behavior that you think justifies special or additional note beyond what is verbally conveyed based on the rubric form assessment.

Image for steps 2.5.a – 2.5.c

2.6 Complete the total score boxes 2.6.a There are no points awarded for Unacceptable ratings, an improvement

plan must be completed.

2.6.b Count the number of boxes circled in the Expected column.

2.6.b.1 Add 5 points for each box circled.

2.6.b.1 Write total in the Score box in the Expected column.

2.6.c Count the number of boxes circled in the Exceptional column.


2.6.c.1 Add 10 points for each box circled.

2.6.c.2 Write the total number in the Score box in the Exceptional column.

Image for steps 2.6.a – 2.6.c

2.7 Complete ELP Academic Improvement Planner (ONLY if student has been marked as Unacceptable in the Academic criteria category) – see Appendix XX (not included in this document) to reference the form.

2.7.a Check the box that best reflects the specific area of needed improvement

2.7.b Provide comments that support the academic improvement plan.

2.8 Complete the ELP Skill Improvement Planner (ONLY if student has been marked as Unacceptable in the Skill criteria category; if student has an Unacceptable in Academics refer to step/decision table 2.3.c) – see Appendix XX (not included in this document) to reference the form. 2.8.a List the competency groups and/or certifications that need improvement,

in terms of completion progress.

2.8.b Check the box(s) that apply for the Required Probationary Action Steps for Improvement.

2.8.c Provide comments that support the skill improvement plan.

2.9 Complete the ELP Behavior Improvement Planner (ONLY if student has been marked as Unacceptable in the Behavior criteria category; if student has an Unacceptable in Academics OR Skill refer to step/decision table 2.3.c) – see Appendix XX (not included in this document) to reference the form. 2.9.a Based on the assessment on the ELP performance rubric form, circle the

correct area of Unacceptable behavior.

2.9.b Write an improvement/assignment due date 1 week from the date of the evaluation.

2.9.c Provide comments that support the skill improvement plan.

M Caution: These forms can be subpoenaed so ensure your comments are professional, truthful and that you check for spelling and grammar (google search words/grammar if you have doubts).

' Hot Tip:

o After completing the assessments and forms for all the students on your list, try to prepare for the face-to-face meetings by either practicing with other instructors or mentally envisioning going through the meetings.

o Put yourself in the student’s shoes and think about how you would want feedback – either for Exceptional, Expected or Unacceptable behavior, but also remember


that the student is accountable for their ratings and they have to make the choice to improve.

3.0 Set-up interview site (A job aid can be used).

3.1 Arrange chairs to facilitate face-to-face contact.

4.0 Debrief student using ELP performance rubric. Use the following table (this step is outside of the team’s scope for a 2-3 hour training session).

If Then

If exemplary student Typically, the student nods in agreement and signs the assessment form. The session is short as there are typically few to no questions.

If average student Typically, the student is in agreement and may have a few questions on how to improve performance. The session is average in length.

If weak student The instructor must be able to discuss the student’s performance and provide recommendations on how to improve performance. The supporting assessment forms must substantiate a report of poor performance.

The instructor will be expected to exchange more information and as a result this session may take longer than the average session.

4.1 Present the feedback in a non-threatening way

4.2 Specify details when giving examples.

4.3 Ask open-ended questions.

4.3 Normalize the conversation at the end of the session.

' Hot Tip: 1. Remain calm during the feedback session.

2. When meeting with a disappointed student:

o Listen attentively and patiently to the student’s concern

o Remain calm

o Place the “burden of proof” on the student. Do not placate the student but rather ask the student to explain how their work fulfills the requirements of grading criteria

o Be future oriented

o Let the student know that you are there to aid in their development.


(3-5 things) of What Is Critical, Difficult, and Complex 1. Completing the behaviors criteria section of the performance evaluation forms

consistently and accurately. a. Interpreting documentation in the student file that was not submitted by the

evaluating instructor; b. Consulting with other instructors to accurately assess students’ performance.

2. Circling the correct behavior criteria rubric boxes based on the information provided in the student’s file, information provided by the familiar instructors, and common examples of behaviors for each situation. This is very difficult for novice instructors.

3. Adequately compiling notes, including student-specific examples for each behavior criteria rubric box rating based on instructors own observations, file documentation or consultations with other instructors, for each assigned student within the time frame provided.


Learner Analysis Effective Feedback for Performance Evaluations and Beyond

Brief Description of the Learners

The learners are Training Specialists at Northwest Lineman College (NLC), a for-profit vocational college. The Training Specialists

are located in Meridian, Idaho; Oroville, California; Denton, Texas; and Edgewater, Florida. At least 15 of the students are

new/novice instructors.

The learners are qualified journey-level lineworkers who have been trained and certified as Training Specialists. These learners are

all men who have a significant amount of experience in the trade with 59% of learners having over 10 years in the trade. Their level

of education is: 45% have high school/GED; 37.5% have completed vocational or technical training; 4.2% have an associate degree;

and 12.5% have completed a Bachelor’s degree. Forty-six percent of learners have 4 years or more at NLC, 25% have 1-3 years

with NLC, and 29% have less than 1 year at NLC. Currently, new-hire Training Specialists complete a 40-hour training program

introducing education and training principles and practices that will get them started teaching in the classroom and field. After the

initial certification training, all Training Specialists are required to complete 40 hours of continuing education a year that builds on the

introductory content of the initial certification.

The learners have stated in interviews and surveys that verbal and written feedback is an essential part of the education they provide

to future linemen. The Training Specialists have indicated that processes and procedures have changed over time and that it is

difficult to truly know a student’s knowledge, skill, and behavior if they don’t interact with the student on all levels in which they are

evaluated. Specifically, they indicated that completing the behavioral form and providing feedback to weak students was the most

difficult.

Sources and Methods

Data Sources Data Collection Method(s)

Training Specialist, exemplary performer - SME Phone interview

Face-to-face interviews

Training Specialist, novice performer - SME Phone interview

Training Specialists Surveys

Directors / Leaders Survey

NLC Training Specialists and staff On-site observations


Human Resources Reporting Extant data

Data-Gathering Instruments

Data was collected using the following instruments:

• Phone Interview questions were based on the Updated BEM and were used when interviewing both an exemplary instructor

and a novice instructor. The questions generated responses that provided information on the learner, their work, and their

environment.

• Face-to-face interviews were used to gather more in depth information about the learner, work processes, and the

environment.

• Observations were conducted informally.

• Extant data was provided by HR on the demographics of the learners. • Surveys were based on the Updated BEM to gain an understanding of the learner, the work, and the environment. Survey

questions and responses also served to confirm findings from the phone interviews. Surveys included:

o Training Specialist Learner Analysis Survey - The survey, conducted 12 October 2016, sampled 24 Training

Specialists and had a 62.5% return rate. See Appendix C for compiled raw data.

o Performance Evaluations for the ELP Student Survey - The survey, conducted 29 September 2016, sampled 24

Training Specialists and had a 75% return rate.

o ELP Student Performance Evaluation Survey - The survey, conducted 29 September 2016, sampled 10 directors and

had an 80% return rate.


Findings and Implications

Relevant Skills

Learner Characteristic Findings: What you found out about each characteristic

Data Source/Method: List a data source/method for each finding

Instructional Implications: Ideas for the instruction in the form of

“therefore” statements Prior Knowledge—what

the target population

already knows and does

All learners have received some

training from Grid U. 20% are

either neutral or disagree that this

training has helped to develop

knowledge and skills.

Survey - Training Specialist

Learner Survey

Performance Evaluations for the

ELP Student Survey

Therefore, training should address

the areas that are not previously

covered in Grid U training and/or

should cover what is difficult or

complex.

There is a wide range of prior

knowledge when completing

student performance evaluations.

Experience ranges from

completing nil terms to 10 or

more terms.

Learners have conducted student

performance evaluations:

• Nil terms: 12.5%

• 1-3 terms: 6.25%

• 4-6 terms: 12.50%

• 7-9 terms: 6.25%

• 10 or more terms: 62.5%


ELP Student Survey

87.5% of Training Specialists

surveyed have conducted formal

student performance evaluations.

Therefore, to narrow the scope of

the training, the instruction should

focus on one particular group of

learners, the novice learner.

A potential consequence of

training all students together is that

the experienced learner will be

bored and may develop a negative

attitude toward the training.

Conversely, if training is focused at

the more experienced learner, the

training may be too challenging for

the novice learner. The novice

learner requires opportunities for

success if confidence is going to

be developed.

Prerequisite skills that

learners should possess

before they can

Learners have conducted student

performance evaluations:

• 4-6 terms: 12.50%


ELP Student Survey

Therefore, these experienced

individuals will not be required to

attend class with the novice. The


challenge the training. • 7-9 terms: 6.25%

• 10 or more terms: 62.5%

Learners know the expected level

of performance when conducting,

and providing effective feedback

during evaluations:

• 44% agree

• 44% strongly agree

learners who have conducted

assessments for at least 4 terms

and who know the expected level

of performance when conducting

and providing effective feedback

during evaluations will be able to

challenge the training.

The super-performers may coach

the novice learners. The super-

performers will also aid in

determining the level of mastery.

Targeted Skills—what the

target population will

specifically need to

learn—and what’s critical,

difficult, or hard about

that

60% of all respondents indicated

that the behavior component of

the assessment is the most

difficult to complete because it

requires complex decision

making.

13% state that it’s difficult to

assess student performance in

the behavior category using the

ELP performance rubric for

familiar students.


Learner Survey

Interview with novice instructor

Therefore, the scope of this

training will focus on the behavior

component of the ELP

Performance Rubric.

Additionally, the learner will be

provided with a behavior

performance decision-table job aid

that the learner will use during the

training and later on the job.

Therefore, using the ARCS model,

the learners should practice

assessing behavior in a safe

learning environment until they feel

confident that they can do the job.


Assessing student files must be

conducted within 20 minutes.

Feedback from respondents is

that this is difficult to do.


Learner Survey


Therefore, during the application

phase, the training will use

scaffolding so that the learner can

achieve the 15-20 minute limit.

The first and second practice

activities will have a 25 minute

time limit to complete one

assessment, the third and fourth

practice activities will have a 30

minute limit to assess two

assessments.

The learner will be given 30

minutes to complete the

assessment activity.

Over 50% of respondents

indicated that obtaining enough

information about unfamiliar

students in order to accurately

assess performance is difficult.


Learner Survey


Therefore, during the application

phase, the learner will be provided

4 practice scenarios to assess

behavior performance. The

Student File Packet for each

scenario will be missing relevant

behavior information. The learner

will have to identify the missing

data on behavior performance,

identify where to obtain missing

data and will then have to obtain

the missing data prior to assessing

the behavior level of performance.

Ideally, SMEs will be available to

assume the role of a source of

missing information. This will

ensure that valid and reliable data

is provided to all learners. The

SMEs may also be able to model

how to provide information.

Respondents indicated that Therefore, the training will


training would be most helpful for:

• Training on how to manage

weak students and what

happens if they don’t improve

• Termination

emphasize an objective and

systematic approach to the

assessment.

A flowchart will be provided as part

of the job aid to guide the learner

in completing the assessment.

Provide the learner with a

Worksheet job aid for taking notes

on the factors that lead to

decisions on the behavior

performance level.

Respondents self-reported that

their own level of performance in

assessing student performance in

the behavior categories on the

ELP performance rubric was as

follows:

• 13% rate their performance to

be excellent;

• 60% rate their performance

as good; and

• 27% rate their performance

as average.

This reflects that there is a wide

range of experience and self-

reported expertise amongst the

Training Specialists.

Therefore, focusing on only the

novice will ensure that those

Training Specialists with over 10

terms are not subjected to the

same training as the novice. It is

assumed that the experienced

performers self-rated as excellent

and good.

Relevant Contextual Factors


Data source/method List a data source/method for

each finding


“therefore” statements Orienting Context—

Learner motivations

related to the job and

any corresponding job

Reception to feedback:

• 60% of Training Specialists

indicate that they are open to

receiving feedback.

Survey- Training Specialist

Learner Survey

Therefore, the instruction should

provide appropriate feedback.

Experienced Training Specialists

will be able to challenge the




each finding


“therefore” statements training • 33% indicate that they strongly

agree that they are open to

receiving feedback on

assessing student

performance.

• 7% disagree.

training. Super-performers will be

encouraged to coach the less-

experienced.

Therefore the instruction will

provide the novice learner with

feedback on one’s performance.

The novice will also be able to self-

assess by noting the time it takes

to complete the performance

activities, as well as, their

improvement to identify missing

pieces of data.

Relevance of assessing student

performance and providing

feedback during formal student

evaluations as part of the job:

• 86% indicated that they agree

or strongly agree that it is

relevant.

• 7% are neutral

• 7% strongly disagree

Survey- Training Specialist

Learner Survey

Therefore, the instruction should

be focused in the application

phase where feedback skills can

be developed during realistic and

varied performance assessment

scenarios.

Therefore the learner will be

directed to take notes on key

behavior factors that led to their

decision on performance level.

This will aid in providing feedback

on the ELP Behavior Improvement

Planner.

72.2% of Training Specialists rated

their comfort level with providing

feedback to students that need

improvement in the behavior

Therefore to maintain Attention,

Relevance, Confidence, and

Satisfaction, the training should be

directed toward the novice who




each finding


“therefore” statements phase as ‘Excellent’.

need to develop their confidence

level.

Instructional Context—

Any factors in the

learning environment

that may affect learning

Learners indicate that they prefer:

• 47% group classroom

environment

• 20% e-learning programs

• 33% one-on-one classroom

• 33% self-directed/self-paced

• 74%-realistic work environment

with mock scenarios to practice

and learn from mistakes


Learner Survey

Therefore, instruction during the

application phase should be based

on realistic scenarios involving a

range of borderline acceptable to

unacceptable behaviors, with

realistic Student File Packets, for

the learner to assess.

The learning environment could be

a traditional classroom setting,

online, or OJT. The environment

will be determined by factors such

as time and cost.

The learners will be able to

complete practice activities with

diminished coaching under

realistic time constraints.

The learning environment for a

Training Specialist is safe and

positive:

• 80% agree or strongly agree

• 20% are neutral or disagree


Learner Survey

Transfer Context—Any

factors related to the

workplace that could

facilitate or hinder

application of learned

skill on the job

Barriers to applying what is

learned in Grid U sessions to the

workplace include:

• 33% not enough time to apply

what I have learned

• 7% It’s too hard to apply what I

have learned to my work.

• 33% It doesn’t seem like

anyone else really applies what

is learned to the job.

• 7% Not enough support or


Learner Survey

Therefore, during the integration

phase, the learners will prepare a

plan for using the new knowledge

and skills on the job. Emphasis will

be given on how the job aid can

assist the learner when completing

the ELP Performance Rubric on

the job.

Provide Training Leads with the

job aid and worksheet for




each finding


“therefore” statements reinforcement from colleagues

or leaders to apply what is

learned to my job.

• 40% indicate that there are no

barriers.

observing future performance

evaluations.

Mode of Delivery

Delivery Mode (select one or more) Rationale

ü Instructor-led training. Can include:

ü Traditional classroom

ü Virtual classroom

ü Self-paced training. Can include:

o Print-based workbooks

ü Online/eLearning that learners complete individually

o Structured on-the-job training

o Flipped classroom

The primary delivery mode for this training will be instructor-led training

at the Boise campus. This mode was selected as it meets the criteria

of transferring the knowledge and skills to the new instructor, cost, and

time factors.

New NLC instructors attend a Grid U training program at the Boise

campus before assuming their duties at their home campus. It has

been identified that there is time in the schedule to add this training

component. Thus, there are no additional costs for travel and lodging.

The traditional classroom with instructor-led training will enable the

training to focus on the targeted tasks that the learner has identified as

being critical and difficult:

• Obtaining information on unfamiliar students

• Preparing feedback for weak students

• Completing the assessments within 15-20 minutes

We considered other modes of delivery that would support learners

using realistic scenarios for practice. However, we determined that by

narrowing the scope of this training to the less than 15 novice learners,

the time and costs involved in providing an eLearning program would

be too prohibitive for any immediate training.

In creating the training for an instructor-led training program, we

recommend that consideration be given to making this an e-learning

program in the future, should the number of potential learners


increase.

We also recommend that experienced Training Specialists and

Training Leads provide coaching and mentoring to the novice Training

Specialist.

Job-Focused objectives Providing Feedback for Performance Evaluations and Beyond

Critical Tasks that the Training Will Address Task 2.4

4. Completing the behaviors criteria section of the performance evaluation forms consistently and accurately.

a. Interpreting documentation in the student file that was not submitted by the evaluating instructor;

b. Consulting with other instructors to accurately assess students’ performance.

5. Circling the correct behavior criteria rubric boxes based on the information provided in the student’s file, information provided

by the familiar instructors, and common examples of behaviors for each situation. This is very difficult for novice instructors.

6. Adequately compiling notes, including student-specific examples for each behavior criteria rubric box rating based on

instructors own observations, file documentation or consultations with other instructors, for each assigned student within the

time frame provided.


Job-Focused Instructional Objectives # Performance (include TA #) Conditions on the job Criteria on the job

1 (2.4 ) Assess student behavior

performance

Cues • Formal student performance

evaluation process schedule

(preparation occurs Wednesday of weeks 6, 10, and 15, the day prior to the formal student performance evaluations which are scheduled to occur on the Thursday of weeks 6, 10, and 15 of the ELP – program is 15 weeks total);

• Receives the Student File Packet

and student list to be evaluated

from the Academic Assistant.

Resources • Student File Packet;

• Worksheet (a job aid used to collect missing information and compile notes for the Student File Packet);

• ELP performance rubric form

• Academic assistant

• Other instructors

• Mission window policy

• Confidentiality policy

• Uses worksheet job aid to gather

data;

• Verifies that Student File Packets

are complete;

• If incomplete, identifies the

missing information (e.g. missing behavioral spot evaluation forms, forms lack enough detail, and/or instructor has not worked directly with the student in the field, classroom and/or behaviorally);

• Consults with the academic

assistant and/or other instructors

to collect missing information and

complete the Student File Packet.

• Consultation conversations:

o Are focused on the student

behavioral issues in

relation to successful

completion of the ELP;

o Compliant with NLC’s

mission window

o Compliant with NLC’s

confidentiality policies;

• Behaviors criteria ratings on the

ELP performance rubric form are:

o Align with the information

in the Student File Packet

and worksheet job aid;

o Marked based on

information in the Student

File Packet and the

worksheet job aid;


# Performance (include TA #) Conditions on the job Criteria on the job o Objective;

o Legally defensible;

• Complete ELP performance rubric

forms;

• If necessary, complete behavior

improvement planners;

• Complete worksheet fields.


Performance Assessment Instrument Providing Feedback for Performance Evaluations and Beyond Objectives

Performance Conditions on the job Criteria on the job

(2.4) Assess

unacceptable

student

behavior

performance

Cues • Formal student performance evaluation

process schedule (preparation occurs Wednesday of weeks 6, 10, and 15, the day prior to the formal student performance evaluations which are scheduled to occur on the Thursday of weeks 6, 10, and 15 of the ELP – program is 15 weeks total);

• Receives the Student File Packets and

student list to be evaluated from the

Academic Assistant;

• Receives a Student File Packet that is

missing information (e.g. existing data is missing from Student File Packet or existing data lacks detail/wasn’t filled out by evaluating instructor, and/or instructor has not worked directly with the student in the field, classroom and/or behaviorally).

Resources • Student File Packet;

• Worksheet (a job aid used to collect missing information and compile notes for the Student File Packet);

• ELP performance rubric form

• Academic Assistant

• Other instructors

• Mission window policy

• Confidentiality policy

• Uses worksheet job aid to gather data;

• Uses worksheet job aid Table A to verify that Student

File Packets contain all existing written documentation;

o If not collects any existing written documentation

missing from Student File Packet from the

Academic Assistant;

• Uses worksheet job aid Table B to determine if the

existing written contains enough information to make a

decision on rating behavioral performance;

o If not, uses worksheet job aid Table C to

§ consult other instructors to collect

information;

§ make substantiated decisions on

behavioral ratings;

• Consultation conversations:

o Are focused on the student behavioral issues in

relation to successful completion of the ELP;

o Compliant with NLC’s mission window

o Compliant with NLC’s confidentiality policies;

• Worksheet job aid is complete;

• Transfers rating decisions from the worksheet job aid to

the behaviors criteria section on the ELP performance

rubric form.

• Ratings are:

o Categorized based on criteria indicators (e.g.

safety, attitude, housekeeping, excellence,

integrity;

o Substantiated based on information in the

Student File Packet and the worksheet job aid;


Performance Conditions on the job Criteria on the job o Objective;

• ELP performance rubric forms, and if necessary

behavior improvement planners, are complete.

Type of performance assessment and rationale Type of assessment Rationale X Product assessment X Process assessment

As indicated in the task analysis, completing the behaviors criteria section of the ELP performance rubric

requires analyzing information to determine completeness, collecting additional required information and

then making complex decisions to accurately assess student behavioral performance on the ELP

performance rubric form.

In order to assess learner performance in assessing unacceptable student performance, team NLC

designed an assessment with both process and product assessment elements. The team chose both

elements because, while the majority of the objective criteria can be measured by reviewing the finished

job aids (i.e. the finished products/product assessment) during or after the assessment, the consultation

conversations part of the objective criteria is more process oriented and needs to be observed during

the assessment.

Because the process assessment element is only one part of the overall objective, which includes the

consultation conversations between the evaluating instructors and other instructors to gather information,

the team feels both process and product elements can be included in one assessment without creating a

time slip within the instructional plan.

Assessment instrument Team NLC instructor will utilize a checklist to assess instructor’s performance in rating unacceptable student behaviors. The checklist will

indicate whether or not the instructor utilized consultations with other instructors and the job aids (the worksheet and ELP performance rubric

form) correctly and completely to rate student behavioral performance1.

F 1

A facilitator and a group of exemplary performers (SMEs) will use the checklists to assess performance during the final assessment, but will also use the checklist to guide feedback

and error detection during the practice exercises.


Mastery level The process tasks are critical to completing the behaviors criteria section of the ELP performance rubric correctly and reliably and there are

a variety of situations that come up when assessing unacceptable performance therefore Team NLC instructor will utilize one assessment

with two Student File Packet scenarios to measure mastery2.

A facilitator and 1 or 2 exemplary performers (SMEs) will assess instructor performance using a checklist. Some of the SMEs will also act as

the Academic Assistant or other instructors within the scenarios when information must be collected via consultation. The instructors will

have one attempt at each Student File Packet scenario in the assessment and must successfully complete both with a 90% or higher rating

on the checklist. If the instructors/learner fail one or both Student File Packets, additional instruction and practice with a facilitator and/or

SME will be arranged at a later date with future assessments to follow.

Job Aids See Appendices:

o Appendix E: Flowchart job aid/Advance Organizer

o Appendix E: Worksheet job aid

o Appendix XX (not included in this document): ELP Performance Rubric

o Appendix XX (not included in this document): Behavior Improvement Planner

Performance Assessment

SET UP

The performance assessment will include 2 Student File Packet scenarios for the learners to use to demonstrate desired performance and

mastery of the performance objectives.

The learners will have the following resources available:

o 2 incomplete Student File Packets with indications of familiarity with student noted.

o Flow chart and worksheet job aid

o ELP Performance Rubric and Behavior Planner

o Instructor to consult if needed

o 30 minutes to complete the assessment

F 2

The team will also utilize 4 practice exercises to aid in achieving mastery.


The facilitator or SMEs will use the assessment checklist to assess the instructor’s performance and will not provide coaching or feedback

during the assessment. The facilitator and SMEs will have the following resources:

o Scenario Handout and Instruction sheet

o Instruction sheet for assessor/observer

o Instruction sheet for consulting instructor

o 1 specific scenario card for each Student File Packet for facilitator or SME acting as consulting instructor for the scenario (if

they are different)

INSTRUCTIONS AND SCENARIOS

Instructions for Facilitator and/or Subject Matter Expert (SME – exemplary training specialists): The Facilitator will

1. Separate the learners into two or more groups to be assessed in different classrooms/rooms.

2. Depending on class size, ensure that no more than 4 (individuals are in a room and ensure there is at least two SMEs per classroom

group. The Facilitator may be a SME for the group he/she is observing.

3. Ensure that each individual is located in separate corners of the room to avoid excessive disturbance/listening-in.

4. Read the learner instructions and inform of resources and time limitations. Indicate who the “Instructor” and/or “Academic Assistant”

is for the group (you can have two SMEs, but they both should play either part to avoid wait times).

5. Remind the learners that the assessment checklist used during practice activities will be the same instrument used to assess their

performance on the assessment.

6. Provide the scenario cards to each SME (they are pre-trained/practiced on the cards) or keep for your own use.

The SME will:

o Use the scenario cards to provide missing existing documentation/information as noted and provide answers to questions asked

based on the information provided in the scenario cards. Provide answers verbally to the learner.

Scenario Cards: • Student File Packet # 1 has unacceptable excellence, will be missing two pieces of information related to attitude and excellence. • Student File Packet # 2 has unacceptable housekeeping, will be missing three pieces of performance information, safety, attitude,

and housekeeping. Instructions for Learner: You will receive 2 Student File Packets to review. Use the resources provided to complete the ELP performance rubric, and improvement

planner if necessary, for each student.

Resources:

• Student File Packet # 1 and scenario card


• Student File Packet # 2 and scenario card

• Flow chart and worksheet job aid

• ELP Performance Rubric and Behavior Planner

• “Other Instructor” and/or “Academic Assistant” to ask for missing information if needed (other Training Specialists or Training Lead in

the assessment room)

• 30 minutes to complete the assessment

The training Facilitator and “Instructor” and/or “Academic Assistant” will not provide hints or coaching as you progress through the

assessment. The Facilitator will designate who will act as the “Instructor” and/or “Academic Assistant” for you to consult as necessary. The

Facilitator will observe and evaluate your performance based on the Assessment Checklist. If the Facilitator will also act as the “Instructor”

and/or “Academic Assistant” for your group, he/she will inform you prior to starting the assessment.

The Facilitator will provide any additional instructions at the time of the assessment.

Student File Packet Contents and Scenario Cards:

You will receive the following information to complete your assessment:

Student File Packet # 1:

• Contents:

o Good standing academics (mid-B’s in all classroom subjects)

o Good standing skills progress

o Blank absence/tardy counts

o No spot evaluations

• Scenario card:

o You have interacted with this student mainly in the classroom environment. In the classroom, you’ve noticed that the student

is doing ok. He answers questions when asked and is reasonably engaged during classroom activities. You have heard some

students grumble when he is in their group and you’ve overheard some complain about his negativity, but you haven’t

observed anything specific in the classroom environment.

o Because of your limited time in the field prior to this formal student evaluation, you haven’t seen much of the student’s

behavior performance in the field.

Student File Packet # 2:

• Contents:

o Good standing in academics (high B’s and low A’s in all classroom subjects)

o Skills progress has been a little behind (as noted in the spot evaluation)

o Absence/tardy counts are present (2 absences, 1 tardy)

o Student has one spot evaluation for field performance (provided) – no progress/follow up documentation exists


• Scenario card:

o You have interacted with this student in the classroom briefly and don’t really have a good sense for his behavior.

o You’ve interacted with this student in the equipment and underground areas in the field and you’ve noticed he takes more

time than the rest of his crew mates to get through his practice attempts. While his other crew mates are practicing, instead of

observing and taking notes – he is often dazing off into other parts of the field.

You will have 30 minutes to complete both scenarios. You should be able to complete each scenario in 15 minutes or less.

ASSESSMENT CHECKLIST

Task

Number Task Yes No Comments

Upon verifying the Student File Packet is missing information/is incomplete:

1 Uses worksheet job aid Table A to verify that Student File

Packets contain all existing written documentation

2 • If the instructor finds there is missing existing information,

he/she collects the information from the Academic Assistant

(if the scenario observed does not include this circumstance, please write N/A in the comments section and do not include this as a point in the Final Score section of this checklist)

3 Uses worksheet job aid Table B to determine if the existing

written documentation contains enough information to make a

decision on rating behavioral performance (both Student File Packets will NOT contain enough information to make a decision)

4 Uses worksheet job aid Table C to consult other instructors to

collect information

Consultation conversations:

5 • Are focused on the student behavioral issues in relation to

successful completion of the ELP

6 • Compliant with NLC’s mission window

7 • Compliant with NLC’s confidentiality policies


Task


8 Uses worksheet job aid Table C to make substantiated

decisions on behavioral ratings

9 Worksheet job aid is complete

10 Accurately transfers rating decisions from the worksheet job aid

to the behaviors criteria section on the ELP performance rubric

form.

Ratings are:

11 • Categorized based on criteria indicators (e.g. safety,

attitude, housekeeping, excellence, integrity)

12 • Substantiated based on information in the Student File

Packet and the worksheet job aid

13 • Objective

14 Signs and dates the ELP Program Performance Rubric

15 The instructor completes the ELP Behavior Improvement

Planner:

16 • Enters an improvement due date

17 • Writes comments

18 • Signs and dates

19 ELP performance rubric forms and behavior improvement

planners are complete.

Total Yes/Nos

Final Score:

• Each Yes is worth 1 point. Each No is worth 0 (zero) points.

• If the scenario observed includes the circumstance described in task 2,

the total possible points are 19.

• If the scenario observed does not include the circumstance described


Task


in task 2, the total possible points are 18.

• The instructor must achieve a score of 90% or higher to pass the

assessment (to achieve this score the checklist can have no more than

1 No). If the instructor falls below this threshold, refer the instructor to

the Training Lead for further instruction.

Note that based on the results of pilot testing, NLC may want to consider

using weighted scores, giving some items a higher value than others.


Instructional Plan Worksheet A. OveralllengthofthetrainingprogramDesignNotes:

1. Theinstructor-ledtrainingwithscenariocardsis180minutesinlength.2. LearnersareallnoviceTrainingSpecialists.3. Thistrainingoccurswithinthecontextofa40-hourtrainingprogramintroducingeducationandtrainingprinciplesandpractices.4. ExperiencedTrainingSpecialistsorTrainingLeads(SMEs)toprovidecoachingandfeedbacktothenoviceTrainingSpecialist(twoSMEsperclasswillbeeasilyavailableper

client).Howlong(inminutes)istheentiretrainingprogram? 180minutesHowmanyunitswillyouincludeinthetrainingprogram? 1unitB. Usethesetablestobriefly(2–3sentenceseach)describeeachelementofthetrainingprogram

IntroductiontothetrainingprogramHowmuchtime(inminutes)willbeallocatedtotheintroduction?

5minutes

Writeabrief(2-3sentences)descriptionoftheintroduction

Thefacilitatorbeginsby:a) Welcomingtheparticipantstothetraining;b) Introducinghim/herselfandexperienceatNorthwestLinemanCollege;c) IntroducingexperiencedTrainingSpecialists(SMEs)assistinginpracticeexercisesand

assessments;d) Explainingwhythistrainingisimportant,thetrainingobjectiveandhowitsupports

businessobjectives(customersatisfaction);ande) Communicatinglengthoftheprogram,numberofpracticesexercises,andanassessment

uponcompletionofpracticeactivities.Unit1: AssessingUnacceptableBehaviorPerformance Listtheobjective(s)forthisunitPhase Timeallocatedto

eachphaseDescriptionofthephase

Activation 5minutes Thefacilitatorwill:• Remindthelearnerthattheyhavehadsomeexposuretorubricsandtheirroleinevaluationduringtheirinitial

training.• Introducetheadvanceorganizer(Flowchart)andWorksheetasavisualforthenewlearning.• ExplainhowtheELPPerformancerubricandperformanceevaluationsareliketheonestheyusedtogetonthejoband


willbesimilartowhatthestudentswillgetonthejob.• Remember,thatitiscriticaltoassessbehaviorwellbecauseofthedangersinvolvedonthejob.Identifytheissues

nowsothatthelearnercancorrectthem.Ifissuesareidentifiedbutnotpointedout,itisessentiallycondoningthebehavior.Doyouwantpoorperformanceonthejob?

Demonstration 20minutes Thefacilitatorwilldemonstrate3scenarios.• Thefirstscenariowilldemonstratetheassessmentofacceptablestudentbehavior.Thefileismissinginformationon

student’ssafetybehavior.• Thesecondscenariowilldemonstratetheassessmentofunacceptablestudentbehaviorwithaclearviolation.The

studentisshorttempered,argumentative,andrefusedtoparticipateinthecommunityserviceproject.Thefileismissinginformationonthestudent’shousekeeping.

• Thethirdscenariowilldemonstratetheassessmentofunacceptablestudentbehaviorthatisdifficulttoevaluate.Thestudent’sbehaviorisinconsistent.Thefileismissinginformationonthestudent’sbehaviorinintegrity.

Uponcompletionofthethreescenariosthefacilitatorwillcomparethedemonstrations.Duringeachofthedemonstrations,thefacilitatorismodelingthebehaviorofanexemplaryperformerconductingtheassessment.Ineachscenariothefacilitatorwillusetheadvanceorganizerasaguidetowalkthroughstepsofassessingastudent’sbehaviorperformance:

a) UponreceivingaStudentFilePacketverifythatitiscompleteandcontainsinformationonbehaviorcriteriaofsafety,attitude,housekeeping,excellence,integrity.Identifyifanyinformationismissinginformation.

b) Usetheflowchartandworksheetdecisiontablestoobtainandrecordtheinformation.c) Emphasizetheimportanceofkeepingtheconsultationconversationsprofessionalandin-linewiththemission

window/confidentialitypolices.Theconversationistofocusonactualknowledgeofthestudentratherthanassumptions.

d) Assessthelevelofstudentbehaviorperformancee) CompletetheELPPerformanceRubric.

Application Time:140minutestocompletetheactivitiesandassessment.Resourcesrequired:PracticeActivities:12x8StudentFilePackets.Eachfileistorepresentadifferentlearnerat

Numberofpractices:4(110minutes),1attemptforeachpractice.Numberofassessments:1(30minutes),1attempt.PracticeActivity1:StudentFilePacket-studentbehaviorhasaclearintegrityviolation,missing2piecesofinformation(safetyandhousekeeping).PracticeActivitytime:25minutesResourcesrequired:StudentFilePacketsMissingFileInformation:ProblemBehaviors:Environment:Thelearnerswillbeinoneclassroom.Thelearnerswillidentifythemissingpiecesofinformationasagroup.Thefacilitatorwillprovidethemissinginformationtothegroupoflearners.Facilitator:

• WillprovideeachlearnerwithaStudentFilePacketthatismissing2piecesofinformation(safetyandhousekeeping)


eitheranunacceptableorexpectedratingbehavior

necessarytoconductacompleteanalysis.• Thefacilitatorwillcoachthelearnersstep-by-stepthroughtheactivityusingtheadvancedorganizerandworksheetjob

aidtoobtainmissinginformationandmakeratingdecisionstocompletetheELPperformancerubric.• Asagroup,thelearnerswillidentifythemissinginformation,thefacilitatorwillprovidetheinformationtoalllearners.• Asthelearnersworkthroughtheworksheet,thefacilitatorwillprovidefeedbackanderrorcorrectionateachstep.

Learners:Asagroup,thelearnerswillverifythatthestudentfileiscompleteandcontainsinformationforsafety,attitude,housekeeping,excellence,andintegrity.Oncethemissinginformationhasbeenprovided,thelearnerwillindividuallyassesstheStudentFilePacket,usingtheworksheetjobtodeterminethelevelofbehaviorperformance.Thelearnerwill:

• Usingtheworksheetjobaid,analyzethedocumentation;• Takenotes(ontheworksheetjobaid)onthekeybehaviortocompletetheStudentFilePacket;• Makeadecision(rate)onthelevelofbehavior;• CircletheappropriateboxontheELPPerformanceRubric;• SignanddatetheELPPerformanceRubric;• AddcommentsontheELPBehaviorImprovementPlanner.

Debrief:

• Usetheperformanceassessmentchecklisttodiscusscommonerrorsnotedandtipstoavoidcommonerrorsmovingforward.

PracticeActivity2:StudentFilePacket-Student’sunacceptableattitudebehaviorisdifficulttoassess,1pieceofinformationismissing(housekeeping).PracticeActivitytime:25minutes.Resourcesrequired:StudentFilePackets,scenarioinformationcardsFacilitator:

• ProvideeachlearnerwithaStudentFilePacket.• Walkaroundtothelearnersprovidingguidance,feedback,andansweringquestions;• Coachlearnersiftheyattempttomoveforwardwithoutadequateinformation.• Usetheperformanceassessmentchecklisttodiscusscommonerrorsnotedandtipstoavoidcommonerrorsmoving

forward.SMEs(TrainingLeadsorExperiencedTrainingSpecialists)willbeintheclassroomintheroleofAcademicAssistantandotherinstructorstoprovidethemissinginformationwhenthelearnerrequestsit.TheSMEwillmodelbehaviorofprovidinginformation.SMEsmay:

• Providecoachingandfeedbackduringtheactivity;


• Coachlearnersiftheyattempttomoveforwardwithoutadequateinformation.ThelearnerwillassesstheStudentFilePackettodeterminethelevelofbehaviorperformance.TheStudentFilePacketwillbemissinghousekeepingperformanceinformationandwillrequirethelearnertoobtaininformationaboutstudent’sperformance.Theanalysiswillincreaseindifficultyduetothemissinginformationandmorecomplexdecisionsrequiredaboutperformance.Thelearnerwill:

• Verifythatthepacketiscomplete,havinginformationforsafety,attitude,housekeeping,excellence,andintegrity;o Determinethereismissinginformationandusetheworksheetjobaidtoobtainthemissingdatafromsource

ofdata(AcademicAssistantorotherinstructor/SME);• Usingtheworksheetjobaid,analyzethedocumentation;• Takenotes(ontheworksheetjobaid)onthekeybehaviortocompletetheStudentFilePacket;• Makeadecision(rate)onthelevelofbehavior;• CircletheappropriateboxontheELPPerformanceRubric;• SignanddatetheELPPerformanceRubric;• Iflevelofbehaviorperformanceisunacceptable,addcommentsontheELPBehaviorImprovementPlanner.

Debrief:Usetheadvanceorganizerandworksheettoreviewtheprocessofbehaviorassessment.PracticeActivity3:StudentFilePacketx2,

• StudentFilePacket#1hasaclearsafetyviolationwith2piecesofinformationmissing(safetyandattitude).• StudentFilePacket#2hasacceptablebehavior,2piecesofinformationaremissing(safetyandattitude).

PracticeActivitytime:30minutesResourcesrequired:StudentFilePackets,scenarioinformationcardsThefirstStudentFilePackethasaclearsafetyviolationandismissing2piecesofbehaviorinformation(safetyandattitude).ThesecondStudentFilePackethasacceptablebehaviorbutismissing2piecesofbehaviorinformation(safetyandattitude).Thelearnerwillberequiredtoobtainthemissinginformation.Thepracticeactivityhasincreasedindifficultyasthereisanadditionalfilethatmustbeassessedwithinalimitedtimeperiodof30minutes.Environment:Thelearningenvironmentwillchangeasthelearnersaredividedintotwogroups.OneSMEwillbewitheachgrouptoprovidethemissingdata;thelearner’swillhavetoapproachtheSMEforthemissingdata.TheSMEwillusethescenariocardtoverballyprovidethemissinginformation/answerquestionswhenthelearnerrequestsit.IftheSMEsarebusywithanotherlearner,thelearnershouldproceedtothesecondStudentFilePackettodeterminethemissingdata.ThelearnermayrequestthemissinginformationforeitherfileorbothfilesateachinteractionwiththeSME.Thefacilitatorwill:

• ProvideeachlearnerwithaStudentFilePacket;• Walkaroundtothelearnersprovidingfeedbackasrequestedandansweringquestions;• Coachlearnersiftheyattempttomoveforwardwithoutadequateinformation;• Usetheperformanceassessmentchecklisttodiscusscommonerrorsnotedandtipstoavoidcommonerrorsmoving


forward.TheSMEswill:

• Providefeedbackduringtheactivity;• Coachlearnersiftheyattempttomoveforwardwithoutadequateinformation;• Usetheperformanceassessmentchecklisttodiscusscommonerrorsnotedandtipstoavoidcommonerrorsmoving

forward.Thelearnerwill:


ofdata(AcademicAssistantorotherinstructor/SME);• Usingtheworksheetjobaid,analyzethedocumentation;• Takenotes(ontheworksheetjobaid)onthekeybehaviortocompletetheStudentFilePacket;• Makeadecision(rate)onthelevelofbehavior• CircletheappropriateboxontheELPPerformanceRubric.• SignanddatetheELPPerformanceRubric.• Iflevelofbehaviorperformanceisunacceptable,addcommentsontheELPBehaviorImprovementPlanner.

Debrief:Usetheadvanceorganizerandworksheettoreviewtheprocessofbehaviorassessment.PracticeActivity4:StudentFilePacketsx2,Complexdecision-making,difficulttoassess

• StudentFilePacket#1willbemissingtwopiecesofinformationrelatedtoattitudeandexcellence.• StudentFilePacket#2willbemissingthreepiecesofperformanceinformation,safety,attitude,andhousekeeping.

PracticeActivitytime:30minutesResourcesrequired:StudentFilePackets,scenarioinformationcardsTheanalysiswillincreaseindifficultywiththeamountofmissinginformationandthecomplexdecision-makingrequiredtoassessperformance.Thisactivitydrawsontheskillsandperformanceofthelastthreepracticeactivities.Environment:Thefacilitatordividestheclassintotwogroups.EachlearnerhastwoStudentFilePacketstoassess.OneSMEswillbewitheachgrouptoprovidethemissingdata;thelearner’swillhavetoapproachtheSMEforthemissingdata.TheSMEwillusethescenariocardtoverballyprovidethemissinginformation/answerquestionswhenthelearnerrequestsit.FacilitatorandSMEs:

• ProvideeachlearnerwithaStudentFilePacket;• Willprovidelearnerswiththemissingdata,uponrequest.Learnersmayrequestthemissinginformationforoneor

bothofthestudentfiles.(Somelearnerswillhavehadtoreviewthesecondstudentfilewhilewaitingtogainaccesstothemissinginformation.);

• Willnotprovidefeedbackorcoachingduringtheactivity;• Duringdebriefwillusetheperformanceassessmentchecklisttodiscusserrorsnotedandtips.


Thelearnerwill:• Verifythatthepacketiscomplete,havinginformationforsafety,attitude,housekeeping,excellence,andintegrity;

o Determinethereismissinginformationandusetheworksheetjobaidtoobtainthemissingdatafromsourceofdata(AcademicAssistantorotherinstructor/SME);

o NotethatifSMEsarebusywithanotherlearner,proceedtothesecondStudentFilePacket.• Usetheworksheetjobaidtoanalyzethedocumentation;• Takenotes(ontheworksheetjobaid)onthekeybehaviortocompletetheStudentFilePacket;• Makeadecision(rate)onthelevelofbehavior;• CircletheappropriateboxontheELPPerformanceRubric;• SignanddatetheELPPerformanceRubric;• Iflevelofbehaviorperformanceisunacceptable,addcommentsontheELPBehaviorImprovementPlanner.

Debrief:Usetheadvanceorganizerandworksheettoreviewtheprocessofbehaviorassessment.Assessment:StudentFilePacketsx2:Complexdecision-making,difficulttoassess

• StudentFilePacket#1ismissingtwopiecesofinformation(attitudeandexcellence).• StudentFilePacket#2ismissingtwopiecesofinformation(safetyandhousekeeping).

AssessmentTime:30minutesResourcesrequired:StudentFilePackets,scenarioinformationcards,instructionsonhowtoperformtheinstructorrole.ThelearnerwillassesstwoStudentFilePackets.SMEs(TrainingLeadsorExperiencedTrainingSpecialists)willassumetheroleofAcademicAssistantandotherinstructorstoprovidethemissinginformationwhenthelearnerrequestsit.Thelearnerswillobtaintheidentifiedmissingdatafromthecorrectsourceofdata.TheassessmentisacombinationoftheprocessusedtocompletetheassessmentandtheproductofacompletedELPPerformanceRubricandELPBehaviorImprovementPlanner.ThelearnerwillfollowtheprocessoutlinedintheFlowchartjobaid,recordnotesintheWorksheetjobaid,circlethecorrectlevelofperformanceforthestudentbeingassessed,indicatethekeybehavior,andinsertcommentsontheBehaviorPlanner.Environment:Theassessmentwillbeheldintwoclassroomstoaccommodatetwogroupsoflearners.Thelearnerswillbedividedintotwogroupstofacilitatetheassessment.Thefacilitatorwill:

• ProvideeachlearnerwithtwoStudentFilePackets;• Providethelearnerfortherulesoftheassessment,particularlyfortherequestofmissinginformation;• Providethemissinginformationuponrequest.Learnersmayrequestthemissinginformationforoneorbothofthe

studentfiles.(Somelearnerswillhavehadtoreviewthesecondstudentfilewhilewaitingtogainaccesstothemissinginformation.)

• Willnotprovidefeedbackorcoaching.ThelearnerwillassesstheStudentFilePackettodeterminethelevelofbehaviorperformance.



ofdata(AcademicAssistantorotherinstructor/SME);• Usetheworksheetjobaidtoanalyzethedocumentation;• Takenotes(onthejobaidworksheet)onthekeybehaviortocompletetheStudentFilePacket;• Makeadecision(rate)onthelevelofbehavior;• CircletheappropriateboxontheELPPerformanceRubric;• SignanddatetheELPPerformanceRubric;• Iflevelofbehaviorperformanceisunacceptable,addcommentsontheELPBehaviorImprovementPlanner.

Debrief:Eachlearnerwillbedebriefedindividually.ThefacilitatororSMEwillusethePerformanceAssessmentChecklistduringthedebrief.

Integration 5minutes Classroom:Facilitatorwillconsolidatewhatlearnershavecovered,specifically,verifyingthatthepacketiscomplete,howtheworksheetfacilitatesthegatheringofdata,andaidsincomplexdecisionmakingrequiredtocompletetheELPPerformanceRubriconthejob.Integrationactivity:Learnerswillidentifyatleasttwokeylessonslearnedthattheycanapplyonthejob.Facilitatorwillemphasizethefollowingpostcourseinformation:

• NLCexpectsthatlearnerswillusetheseresourcesforassessingstudents.• Afterthetraininghasended,learnerswillworkwiththeirTrainingLead.• TrainingLeadswillbeprovidedwithachecklistforobservingthelearners’futureperformanceevaluations–timewill

dependonELPscheduleforeachcampus.• TheTrainingLeadwillprovidefeedbackandcoachingtothelearnerastheyevaluatefuturestudents.• TrainingLeadshavereceivedcopiesoftheadvanceorganizerandworksheet.

Recommendationsforpost-trainingactivities:

• Brownbaglunchestodiscusswhat’sgoingwellandifanythingneedstobechanged.

ConclusiontothetrainingprogramHowmuchtime(inminutes)willbeallocatedtotheconclusion?

5minutes


Writeabrief(2-3sentences)descriptionoftheconclusion

Facilitatorwillprovidesummaryofthecontentcovered.Advancedorganizerwillbedisplayedasavisual.FacilitatorsandSMEsencouragethenoviceTrainingSpecialiststoutilizetheworksheetjobaidwhenassessingbehavioralperformancewithaStudentFilePacketthatismissinginformationwhentheyareonthejobbecauseitaccomplishestwothings:completingtheStudentFilePacketforfutureperformanceevaluationsandprovidescriticalinformationtoassessdocumentedstudentbehavior.Thefacilitatorwrapsupthecoursebyadministeringalevel1trainingevaluationandindicatesthatalevel3evaluationwillbesentouttoalllearnersandtheirTrainingLeadsseveralweeksormonthsafterthetraining.

PracticeActivity1 PracticeActivity2 PracticeActivity3 PracticeActivity4 Assessment

1

Remindthelearnertoreviewthejobaids.(Flowchartandworksheet)

ü ü

Remindthelearneraboutwhatiscritical,difficult,andcomplex.

ü

Remindthelearnertoreviewtheworksheetjobaid.

ü

Remindthelearnertoreviewtheworksheetjobaid.

Providecoaching(immediateerrordetectionandcorrection)forwhatiscritical,difficult,andcomplex)

ü

Iflearnerattemptstomoveforwardwithoutallrelevantdocument,coachlearnertocheckfordocumentation

ü


ü


Coaching:Stateeachstepinthetasklistbeforeaskinglearnerstocompleteit.

ü ü

Chunking:Provideimmediatefeedbackaftereachdiscretestep.

ü

ü

Usetheperformanceassessmentchecklisttoprovidedelayedfeedback

ü ü ü ü ü


ProblemProgression 1studentfilepacketwith:

• 1clearviolation

• 2piecesofmissinginformation

1studentfilepacketwith:

• Unacceptablebehaviordifficulttoaccess

• 1pieceofmissinginformation

2studentfilepacketswith:

• 1clearviolation

• 1acceptablebehavior

• 2piecesofmissinginformationinbothpackets


• Complexdecisionmaking

• 2piecesofmissinginformationinonepacket

• 3piecesofmissinginformationintheotherpacket


• Complexdecisionmaking

• 2piecesofmissinginformationineachpacket


Team Project Status Review # 1

Project Name:

NLC Instructor

Date: 8 September 2016

Report

Author:

Kara Rutherford

Project Team

Members:

Beverly Naylor

Kara Rutherford

Tammy Wheeler

Overall Project Status in Relation to Schedule

Project Status Summary: X Green ____Yellow ____Red

Indicate the extent to which the team believes the ID project is producing quality work on

schedule.

• What parts of the project has the team completed?

Team Charter

Project Description – including project schedule

• What parts of the project are in progress?

Coursework for performance/cause analysis

• What parts of the project will the team work on next?

Performance analysis

Task analysis

Actual or Potential Roadblocks • What obstacles has the team run into so far and how did you work through or around them?

We had some trouble coming up with an initial meeting time that worked for all of us

considering the Labor Day holiday. We worked around it by accommodating schedules to let

members come in a little late with a review of the discussion as well as setting a date and

time for weekly meetings that will work for all of us going forward.

• What potential obstacles do you see related to what you will be working on next?

Since this is our first status and weekly assignment, we are not sure exactly what to expect,

but managing project time, work time, and family time can be an obstacle in completing our

next part of the project.

We don’t expect any obstacles in getting information from the client.

• What, if any, help will you need to overcome the obstacle(s)?

We don’t expect any outside help will be necessary to overcome potential time obstacles or

getting information from the client. We plan to use our weekly meeting to set realistic tasks

and timelines based on our weekly schedules and will evaluate how well our plans worked

prior to setting the next week’s schedule.


Team Work Successes • What aspects of working as a team have gone well?

We are all very open and active communicators and listeners so we have been able to start

off running. During our initial meeting we used our agenda to cover the following points:

o Meet and greet- this gave us the opportunity to identify other key priorities that we

are facing.

o Personal expectations for the course;

o How we will manage any conflict that may arise over the duration of the project; and

o How we will manage and coordinate weekly meetings. We have a schedule that

identifies who will lead the meeting and who will take minutes. An agenda will be

distributed prior to a meeting. Minutes will be posted following the meeting to ensure

that all members are aware of the team’s expectations of them.

• What will the team do to build on those successes as the project continues?

We will commit to attending all of our weekly meetings and to be honest with challenges we

encounter as well as to be graceful in giving and accepting constructive feedback on work

provided.

Team Work Improvements • What aspects of working as a team could the team improve upon (roles, processes, tools,

etc.)?

It’s a little early to tell yet, but in upcoming project work, we need to set definitive task

assignments for each team member and/or for group task assignments in our weekly

meetings. With the short week and not being able to connect everyone for the full hour, we

were a little rushed at the end of the meeting.

• What steps will the team take to improve these aspects? What, if any, help will you need?

We plan to address this in our weekly meeting agenda and no outside help is needed right

now.


NLC Instructor Team Project Status Review # 2

ProjectName:EffectiveFeedbackforPerformanceEvaluationsandBeyond

Date: 30September2016

ReportAuthor:BeverlyNaylor

ProjectTeamMembers:

KaraRutherfordBeverlyNaylorTammyWheeler

Overall Project Status in Relation to Schedule ProjectStatusSummary:___Green____Yellow____Red

IndicatetheextenttowhichtheteambelievestheIDprojectisproducingqualityworkonschedule.• Whatpartsoftheprojecthastheteamcompleted?

ProjectDescriptionProjectStatusReview1TeamCharterPerformanceAnalysisPass1

• Whatpartsoftheprojectareinprogress?PerformanceAnalysis-GuidedbytheBEM,theteaminterviewedanexemplaryinstructorandanoviceinstructor.Asurveywasthensentoutto34trainingspecialists(24instructors,10managers)andhada47%responserate.Inaddition,theteamhasreviewedtheNLCwebsiteandtheElectricalLineworkerProgramPerformanceRubric.TaskAnalysis–Theteamhasmetwithanexemplaryinstructorandanoviceinstructorinordertoidentifythetasksinvolvedwithprovidingfeedback.TheteamhasreviewedtheElectricalLineworkerProgramPerformanceRubric,andotherNLCdocumentation.

• Whatpartsoftheprojectwilltheteamworkonnext?LearnerAnalysis

Actual or Potential Roadblocks • Whatobstacleshastheteamrunintosofarandhowdidyouworkthroughoraroundthem?

Theteamlostonemember.Tomanagethisloss,wereallocatedthetasksthathadbeenassignedtothatindividual.Theteamcharterhasbeenamendedtoreflectthis.

• Whatpotentialobstaclesdoyouseerelatedtowhatyouwillbeworkingonnext?

Wecurrentlydonotforeseeanypotentialobstacles.

• What,ifany,helpwillyouneedtoovercometheobstacle(s)?

Therearenoobstaclesatthistime.

Team Work Successes • Whataspectsofworkingasateamhavegonewell?

Theteamhasadedicatedmeetingtimetodiscussthetasksthatneedtocompletedforthatweek.Inaddition,theteamhasdemonstratedflexibilityinitsabilitytoadjustmeetingtimes.Theweekof26September,theteammettwicetodiscussthedeliverablesdueon30September.Thereiseffectivecommunicationwithquickresponsetimes.Emailsareexchangedthroughouttheweektodiscusspointsandsharework.Theteammaintainsasenseofhumor.

• Whatwilltheteamdotobuildonthosesuccessesastheprojectcontinues?


Tobuildonthesesuccesses,theteamwillcontinuetomaintainitsdedicatedmeetingtime,itsflexibility,itsfrequencyofcommunicationandsenseofhumor.

Teamwillmodifyandadjustmilestonescheduleasperclientrequest.

Team Work Improvements • Whataspectsofworkingasateamcouldtheteamimproveupon(roles,processes,tools,etc.)?

WhilewediscussourexperienceswithourworkplacesandotherOPWLcourses,we’veidentifiedthatweshoulddiscussthecoursereadings,discussions,andcontentinrelationtoourprojectaswellasinourroundtablediscussionsduringourweeklymeetings.

• Whatstepswilltheteamtaketoimprovetheseaspects?What,ifany,helpwillyouneed?

Theteamcurrentlydoesnotneedanyassistanceintheareaofteamworkimprovement.


NLC Instructor Team Project Status Review # 3

ProjectName:EffectiveFeedbackforPerformanceEvaluationsandBeyond

Date: 11November2016

ReportAuthor:BeverlyNaylor

ProjectTeamMembers:

KaraRutherfordBeverlyNaylorTammyWheeler

Overall Project Status in Relation to Schedule ProjectStatusSummary:___Green____Yellow____Red

WehaveidentifiedthatweareintheYellowleaningtowardsGreenbecausewestillhavethePerformanceAssessmentPass1tosubmit.Whilewehavedevelopedadraftofthisdocument,itisnotreadyforsubmission.WeareconfidentthatwewillbeabletosubmitbothPass1andPass2ofalldocuments.

IndicatetheextenttowhichtheteambelievestheIDprojectisproducingqualityworkonschedule.• Whatpartsoftheprojecthastheteamcompleted?

ProjectDescriptionTeamCharterProjectStatusReview1ProjectStatusReview2PerformanceAnalysisPass1PerformanceAnalysisPass2TaskAnalysisPass1TaskAnalysisPass2LearnerAnalysisPass1LearnerAnalysisPass2ObjectivesPass1ObjectivesPass2InstructionalPlanWorksheetPass1ProjectStatusReview3

• Whatpartsoftheprojectareinprogress?1. PerformanceAssessmentPass1.Draftversion.

• Whatpartsoftheprojectwilltheteamworkonnext?1. InstructionalPlanWorksheetPass22. InstructionalPlanPass13. FormativeEvaluation

Actual or Potential Roadblocks • Whatobstacleshastheteamrunintosofarandhowdidyouworkthroughoraroundthem?

OnechallengethattheteamhasencounterediswithPass1assignments.NLCInstructorstrivestoproduceahighqualityproductandindoingsotheteam’stimelinehasslipped.Currently,thePerformanceAssessmentPlanisinprogress.Wedonotforeseeanyproblemswithhavingthesedocumentsdeliveredbytheduedateoutlinedbythecoursebuthavehadtoadjustourownteamschedule.Anotherchallengeisthatteammemberseachhaveaslightlydifferentinterpretationoftheproject.Theteamhashadtoworktoalignourpersonalvisionoftheprojectintoonesharedvision.While


onlyaminorchallenge,itresultedintheteamcallingforanadditionalmeetingon10NovembertodiscussObjectivesPass2ratherthansubmittingwhatwasbelievedtobeafinalversion.

• Whatpotentialobstaclesdoyouseerelatedtowhatyouwillbeworkingonnext?

Theteamhasidentifiedtheneedtoclearlyarticulateone’svisionoftheprojectinordertodevelopamutuallysharedprojectvision.Maintainingclear,concisecommunicationwilleliminateorreducepotentialobstaclesthatcouldcausethescheduletoslip.

• What,ifany,helpwillyouneedtoovercometheobstacle(s)?

TeamNLCdoesnotcurrentlyneedanyhelptoovercomethedescribedchallenge.

Team Work Successes • Whataspectsofworkingasateamhavegonewell?

TeamNLCInstructorhasadedicatedmeetingtime,usingGoogleHangouts,todiscussthetasksthatneedtobecompleted.Emailsareexchangeddailytodiscusspointsandsharework.Whenemailsarenotsufficienttoresolveanissue,theteamreadilycallsforadditionalsynchronousmeetings.Themostrecentextrameetingwascalledon10November2016.Eachmemberoftheteamisfullycommittedanddedicatedtoaccomplishingthetasks.

TeamNLCInstructorstrivestocompletealldeliverablesinaccordancewiththeirteamschedule.Byaimingtoadheretotheirschedule,thereisflexibilitytoadjustthescheduleshouldadditionaltimebeneeded.TheteamfeelsconfidentthattheywillbeabletosubmitbothPass1andPass2submissionsforalldeliverables.

TeamNLCInstructorhasdevelopedajobaidthatconsistsofaflowchartandworksheet.ThisjobaidhasbenefittedtheteaminthedevelopmentoftheObjectives,PerformanceAssessment,InstructionalPlanWorksheet,andisanticipatedtobeofbenefitfortheInstructionalPlan.

TeamNLChasbecomemorecriticaloftheworkthatisbeingsubmitted.Priortosubmittingadeliverable,wearenowquestioningthedocumentstoverifythatthereisinternalandexternalalignment.

TeamNLCInstructorincorporatesthecourseweeklycontent,discussions,andreadingsintoteamdiscussions.Theresultisthatallteammembersbenefitfromparticularlyworthypointsthatmayhavebeenmissed.

Whilestrivingtoattaintheambitiousprojectschedulethatwedevelopedatthebeginningofthecourse,wehaverealizedthatitisalivingdocumentthatcanbeadjusted.Byaimingtoadheretoit,westayontrack,byacknowledgingthatthescheduleisaguideweareabletofocusontheactualdeliverableratherthanonthedatedue.

• Whatwilltheteamdotobuildonthosesuccessesastheprojectcontinues?

Tobuildontheteam’ssuccesswewillcontinuetomaintainopenlinesofcommunication.Wehavefoundthatdiscussionsconsistingofonlyafewminuteshasbeeneffective.

Tobuildonthesesuccesses,theteamwillcontinuetomaintainitsdedicatedmeetingtime,flexibility,frequencyofcommunicationandsenseofhumor.

Team Work Improvements • Whataspectsofworkingasateamcouldtheteamimproveupon(roles,processes,tools,etc.)?


Wehavefoundthateachofushasspecificrolesthatwehaveadopted.Theprocessappearstobeworkingwell.Theteamcanimprovebysharingmoreoftheinformationthatwehavefromeachofourroles.

• Whatstepswilltheteamtaketoimprovetheseaspects?What,ifany,helpwillyouneed?

Theteammemberswillaskmorequestionsofoneanothersothattheonusisonallmemberstogaintheinformation.


Appendix A: Interview Notes

Frank S. (pseudonym).: Cause Analysis: Variables Questions Typical performer – Novice Environment

1. Information Do the performance

standards make it

clear how you are

expected to provide

feedback?

Are you provided

feedback on this

specific performance?

No, should have had more training on

what to expect and how to answer

questions that they asked me.

No feedback on the specific performance.

2. Resources Are there adequate

resources available to

help you to provide

feedback?

Yes, I think so. But George didn’t know to

ask for it. I believe there are some

available.

3. Incentives Have you received

(are they available)

bonus around good

performance?

Are there non-

monetary incentives

for good performance?

No bonuses. No non –monetary

The only thing I really look for is like an

atta-boy or a good job.

Really enjoys his job and wants to

succeed and if they succeed that is

reward

Individual

4. Motives How do you feel about

providing verbal and

written feedback to

trainees?

I think it’s a good thing because if you

don’t give them feedback they don’t know

what to do to improve or succeed. It goes

both ways.

KRs follow up question -

Don’t see anything getting in the way.

Brand new instructor in the classroom

and asking for feedback from the

students on performance. In the field not

a problem on feedback on what they are

doing because of his experience and

expertise, but in the classroom not so

much

5. Capacity To Manager:

Does the individual

lack the capacity or

ability to learn?


6. Knowledge/Skills Do you believe the

processes and

procedures have

remained consistent?

Do you feel you are

expected to know too

much information?

No.

Yes.

Frank S. (pseudonym).: Task Analysis: Preparing feedback: I got a form from Samantha and had their grades, attendance, rubrics previous, had every

student the entire semester.

I also had the previous knowledge from working with the students over the term.

Drew on what was on paper in front of me, but my semester experience.

Is there anything difficult or complex about form? No, the form is pretty much laid out, but the only think I had issue with is that I didn’t have

enough time per student and if they had questions, I wanted to be able to answer their

questions so I went over the time.

The hardest part of the whole thing was telling the students weren’t going to graduate it. I don’t

want to become hard-hearted and have no intention of becoming hard-hearted.

How do you prepare yourself to do that? Only way I can see it working where it does not keep me up at night, I think long-term when they

leave here, I don’t want them carrying bad habits into the trade that could cause harm to

themselves or others.

Meeting with student: Do you set up in a specific manner? I use the classroom, I am on one side of table and they are on the other. Maintain eye contact

and did best to not twiddle thumbs or be distracted. Have a grown up to grown up discussion

and not treat them like a child.

When you come across something difficult – giving honest feedback. Is it easier to skim over things or how do you ensure that they get all the necessary feedback? For one thing, I am going to ask them do you understand that you will not graduate. And not

condescending, but being honest and straight forward

What about the student that has done exceptionally well? I don’t think so, I do my best to treat all students equally.

After the conclusion of the evaluation process: Any follow up? Saw the students for two days after and would just check in with them.

Can you imagine a scenario where an instructor may not be able to meet that objective?


Yes, they don’t care. Yes

In your experience, are you observing students that are not receiving that type of feedback?

Yes.

Is it that they don’t care or don’t know how to do it? Yes, to they don’t care and yes to they may not know how to do it, they may have passion that

is lacking.

Sam M. (pseudonym). Cause Analysis: Variables Questions Exemplary performer – Veteran

Environment

1. Information Do the performance

standards make it clear how

you are expected to provide

feedback?

Are you provided feedback on

this specific performance?

I think they are verbally communicated and

being part of the organization we are in they

can change readily, so understanding that if

things are written and if there is a plan of

instruction you can follow that and it brings

consistency – the other side of that is that when

it’s time to change you have updates and

managing the change.

Communicated yes, verbally – is it monitored,

checked, feedback – could be better.

Feedback: Seen it happen some, but not as much as we

should. When we train a new instructor there is

some feedback given, but after that we should

be giving feedback more often.

Because feedback should be consistent,

positive and

2. Resources Are there adequate resources

available to help you to

provide feedback?

Yes – the data is there and the data is

processed in order to be accessible and the

instructors are given time to digest the data and

make an accurate counseling based on that

data.

3. Incentives Have you received (are they

available) bonus around good

performance?

Are there non-monetary

incentives for good

performance?

Not monetarily – because there is not really

much feedback on the process so even non

monetarily process isn’t there


Individual

4. Motives How do you feel about

providing verbal and written

feedback to trainees?

I think it’s a very important part of the

educational process. Three ways to learn,

knowledge, skills, behavior. Learning theory is

reinforcement and that comes from counseling

and feedback.


Does the individual lack the

capacity or ability to learn?

6. Knowledge/Skills Do you believe the processes

and procedures have

remained consistent?

Do you feel you are expected

to know too much

information?

Changed. They are important and there has

always been time allotted for formal feedback

and evaluation. Over the 10 years the process

has been polished and been refined to be

better for the student and the instructor.

No, not too much information. The challenge

with knowing for a TS is if you are going to

effectively counsel the student, it’s important to

know the student. If the student is performing

well, it’s easy but if they aren’t you have to get

beyond the data to really understand how to

counsel or encourage or connect with the

student beyond what’s on the data sheet.

You may not interact with the student in all

three areas so it can be hard to counsel

students. I can make circles and boxes but

don’t really know the student.

Sam M. (pseudonym). Task Analysis: How do you prepare for the meeting - walk us through the process?

The way now:

Data is collected the three areas of learning. academic scores, performance in the field –

competency completion criteria, and any counseling sessions that occur for exceptional and

unacceptable behavior.

The information is presented to the instructor for the group of students they are responsible for

counseling. The instructor has time to pour over that and pull the information to put the

information on the forms. The system is set for average and then there is an improvement plan

for those that are not.

In planning for that – there is that tie that needs to be made in knowing the students so without

that it’s hard to counsel the students when you haven’t observed them in all the phases of

education. The students you counsel are the students you work with each day.

When you don’t have a student in the field/classroom.


An instructor should collaborate with other instructors that have worked with that student to get

information to help counsel the student and provide the best feedback.

Does the form get into detail or generic?

It’s a blend. It’s written so there are doors that you could expand on to be specific to the student.

Able to dig down to be specific to student?

If the student has done exceptionally well, there should be documentation in the student file and

you can mark that and then expound on that detail. If those things are in the folder…consistency

is the key and the documentation may or may not be in the folder.

We have a reward and recognition system with stickers on the hard hat and that happens on the

spot – there should be written documentation that matches that in the folder but the practice

now is that stickers are there but no documentation so getting detailed and specific is difficult.

The form is weak in the area for the stickers and the reward/recognition with specific details.

Challenges?

Safeguarding and the allotted time needed to prepare. The hardest ones are the ones that

require improvement. It’s easy for the exceptional student or the middle of the road.

Will some instructors opt for the middle of the road to avoid the difficulty with counseling improvement?

CS: Is there a tendency to want to push them in meets rather than NI – yes because they are on

the border and you aren’t really sure what to say to the student. May need another instructor to

ask for help or get advice.

Objective is good/ subjective is hard

Concrete data is good in the system, but when it comes to the improvement for the behaviors

can be more difficult in the moment.

If training provided, what processes or tasks, steps would be the most beneficial to focus on?

I think basic to intermediate to advanced communication techniques, understanding the

personality types of people. Also effective documentation skills – used to be open for them to

write comments so they system evolved to circles and checkmarks to make it a little more

standardized which has its pros and cons.

If we are sticking with the system we have now, some basic training on what each of those

boxes mean and maybe even a more advanced training on what those boxes mean. When we

bring someone in they get 1 or 2 hours training, but you need to model it, you need to see it,

observe it, so you need to get into those situations for all varieties of students so they can

develop the areas they are weak in counseling.

Assess where they are strong and weak and then assess them in a training environment before

they are doing it in a real situation.

Get foundational training and then it’s thrown in there and then you learn as you go.

Form: What is the most difficult aspect of that form? CS: the behavioral part, the classroom is cut and dry, skills can be more difficult because it’s a

little more subjective, but students will ask what they can do to be exceptional – they students

don’t know, does the new instructor know what to tell them? IF you have been doing it for a

while you will know better what to say. You just have to use your common sense if you don’t

have the experience.


That carries over into behavior – there are a lot more areas to asses. Students will question how

they can do better. If you collaborate with instructor before-hand and then students will ask why

they are not exceptional. They think they are exceptional but they are clearly middle of the road.

So working with instructors to know how to answer that is very difficult. When students ask

questions, the evaluation is harder.

Human nature is to gravitate toward the middle of the road or positive to not have to deal with

the difficult feedback. The documentation that needs to be in place to substantiate behavior that

needs to be improved, if it’s not there, it’s really difficult.

If the instructors are not using the forms between the formal evaluations, you don’t have a

picture of the student in order to give them adequate feedback.

The forms can be legal forms and used for terminations, etc. so the instructor may not want to

write much on the forms to avoid any issues in the future.

Scheduling

Used to be two instructors, but now it’s down to one instructor and you have a group of students

assigned to you.

Changed because of man power. They try to schedule students you know or have worked with.


Appendix B: Chevalier’s Updated Behavioral Engineering Model (BEM) Order Completed for NLC Training Specialist Chevalier’s Updated Behavioral Engineering Model (BEM) Order Completed for NLC Training Specialist

Variables Questions Exemplary performer interview Typical performer interview

Environment

1. Information Do the performance standards make it clear how you are expected to provide feedback?

Are you provided feedback on this specific performance?

I think they are verbally communicated and being part of the organization we are in, they can change readily. So understanding that if things are written and if there is a plan of instruction you can follow that and it brings consistency.

The other side of that is that when it’s time to change you have updates and managing the change.

Communicated yes, verbally – is it monitored, checked, feedback? – could be better.

Feedback:

Seen it happen some, but not as much as we should. When we train a new instructor there is some feedback given, but after that we should be giving feedback more often because feedback should be consistent and positive.

No, should have had more training on what to expect and how to answer questions that they asked me.

No feedback on the specific performance.

2. Resources Are there adequate resources available to help you to provide feedback?

Yes – the data is there and the data is processed in order to be accessible and the instructors are given time to digest the data and make an accurate counseling based on that data.

Yes, I think so. But I didn’t know to ask for it. I believe there are some available.

3. Incentives Have you received (are they available) bonus

Not monetarily. Because there is not really much feedback on the process non-monetary process

No bonuses. No non-monetary.

The only thing I really look for is like an


around good performance?

Are there non-monetary incentives for good performance?

isn’t there. “atta-boy” or a good job.

I really enjoy my job and want students to succeed and if they succeed that is reward enough.

Individual

4. Motives How do you feel about providing verbal and written feedback to trainees?

I think it’s a very important part of the educational process. Three ways to learn: knowledge, skills, and behavior. Learning theory is reinforcement and that comes from counseling and feedback.

I think it’s a good thing because if you don’t give them feedback they don’t know what to do to improve or succeed. It goes both ways.

I don’t see anything getting in the way. As a brand new instructor in the classroom and asking for feedback from the students on performance. In the field, it’s not a problem on feedback on what they are doing because of my experience and expertise, but in the classroom not so much.


Does the individual lack the capacity or ability to learn?

No. No.

6. Knowledge/Skills

Do you believe the processes and procedures have remained consistent?

Do you feel you are expected to know too much information?

Changed. They are important and there has always been time allotted for formal feedback and evaluation. Over the 10 years the process has been polished and been refined to be better for the student and the instructor.

No, not too much information. The challenge with knowing for a Training Specialist is if you are going to effectively counsel the student, it’s important to know the student. If the student is performing well, it’s easy, but if they aren’t you

No.

Yes.


have to get beyond the data to really understand how to counsel or encourage or connect with the student beyond what’s on the data sheet.

You may not interact with the student in all three areas so it can be hard to counsel students. I can make circles and boxes but don’t really know the student.

Effective Feedback for Performance Evaluation Rutherford, Naylor, & Wheeler 80

Appendix C: Survey Questions/Results Survey 1


Survey 2


Appendix D: Extant Data Examples NLC’s mission/vision Northwest Lineman College will be a dynamic, visionary, and leading international educational institution providing the benchmark standard of safety, training, and products that exceed the expectations of customers in the power delivery and related industries. NLC’s core values Northwest Lineman College has an outstanding reputation throughout the power delivery industry, thanks to the high standards of its company, staff, and students. The Core Values and Mission Statement are used as the guiding principles by which NLC conducts training, business, and all endeavors. P.I.E. = PASSION, INTEGRITY, EXCELLENCE* PASSION Love what you do with PERSISTENCE, MOTIVATION, and the ATTITUDE to succeed. INTEGRITY Do the right thing—even when no one is watching—so TRUST, HONOR, and DEPENDABILITY are your hallmarks. EXCELLENCE Innovate always, leveraging ORIGINALITY and TEAMWORK, so your LEGACY of positive contribution to the industry is guaranteed. *(PIE is a mnemonic for the power formula, P (watts) = I (current) X E (voltage), which is a well-known and used formula in the power delivery industry):

Electrical Lineworker Program (ELP) Objective The ELP program objective is to provide training and education that ensures graduates are highly qualified for initial employment, highly successful in completing formalized apprenticeship and training programs, and are prepared to pursue long, rewarding careers as lineworkers.

ELP Performance Rubric and Coordinating Improvement Planners for Academics, Skills, and Behavior (Not included in this document)


Appendix E: Job Aids

Advance Organizer Flow Chart

Rutherford, Naylor, Wheeler November 17, 2016 1

Worksheet Job Aid: Assessing Areas for Behavioral Improvement

Use this worksheet to collect and document data necessary to make performance assessment decisions for the behaviors criteria section on the ELP performance rubric. Note: This worksheet will also provide talking points and examples to discuss during the face-to-face performance evaluations with the student. Complete the steps listed below:

1. Determine if the Student File Packet contains all existing written information to assess behavioral performance: Table A If the Student File Packet: Then: Contains one or more spot evaluations for exceptional behavior and has absence and tardy counts listed

1. Assess the student using the ELP performance rubric form Hot Tip: The Student File Packet is likely complete and can be assessed using the ELP performance rubric form alone. It’s very rare that a student with spot evaluations for exceptional behavior will require additional consultations or information to assess behavior. If this does occur, consult with your Training Lead immediately to aid in completing the assessment.

Contains one or more spot evaluations for exceptional behavior but is missing absence and tardy counts 1. Ask the Academic Assistant to provide the absence and/or tardy counts. 2. Note any counts on the Student File Packet absence/tardy count sheet. 3. Assess the student using the ELP performance rubric form.

Hot Tip: Once you collect the absence/tardy counts, the Student File Packet is likely complete and can be assessed using the ELP performance rubric form alone. It’s very rare that a student with spot evaluations for exceptional behavior will have excessive absence/tardy counts and/or will require additional consultations or information to assess behavior. If this does occur, consult with your Training Lead immediately to aid in completing the assessment.

Is missing any one or combination of the following documentation: • Spot evaluations:

o For probation for violating behavior standards o For probation for not meeting written program requirements

1. Ask the Academic Assistant to check files/records to verify if the documentation exists or doesn’t. 2. Collect any existing documentation not previously provided by the Academic Assistant OR note that the

information doesn’t exist on Step 3. 3. Proceed to Table B under step

Is missing One or both of the following documentation (indicated by a blank space by either count on the absence/tardy sheet in the Student File Packet): • Absence Counts • Tardy Counts

1. Ask the Academic Assistant to provide the absence and/or tardy counts. 2. Note on the Student File Packet absence/tardy count sheet. 3. Proceed to Table B under step 2

2. Review the available behavioral data in the Student File Packet


3. Determine if there is enough existing behavioral data in the Student File Packet to make a decision (rate) the student’s behavioral performance:

Table B If the Student File Packet: Then: Has no existing spot evaluations for any one or combination of the following:

o Probation for violating behavior standards; o Probation for not meeting written program requirements; o Probation for excessive absence/tardy counts.

Use Table C to: 1. Document performance findings/notes for each behaviors criteria for the students you have personally

observed classroom and/or field; 2. Consult with other instructors to document performance findings/notes for each behaviors criteria for the

students you have NOT personally observed in the classroom and/or field. Contains one or more spot evaluations for probations for behavior that you didn’t personally fill out: Use Table C to:

1. Consult with the instructor who did fill out the spot evaluation to document performance findings/notes for the specified behavior criteria on the spot evaluation form.

Continued on page 3


4. Use the following tables to: a. Determine if behavioral issues exist for each behaviors criteria category; b. Document findings and examples; c. Rate student behavior on the ELP based on the findings and examples.

Table C: Safety

Behavior Criteria Category

Behavior Questions Record Findings Note answers below or indicate N/A if the criteria does not require additional information.

Record Examples Note examples of observed behaviors for question 1 and any other questions listed as necessary.

Rating Guidance

SAFE

TY

1. If the student has a spot evaluation for safety, is the student’s performance improving?

a. Yes: Examples b. No: Examples

2. Does the student always wear gaff guards?

3. Does the student always climb with proper gloves?

4. Does the student always wear required personal protective equipment (PPE)?

5. Has an injury or injuries occurred after coaching/feedback from instructor?

6. Are tailboards always completed per NLC standards/requirements?

If the answers to 3 of the 6 questions indicate ongoing poor performance and/or an unresolved spot evaluation(s) exists for safety violations, the student is performing at the Unacceptable level for safety. Otherwise the student is performing at the Expected level for safety.

1. Rate the student’s behavior for Safety using the information documented in the record findings space, the record examples space and the rating guidance information:

a. Circle the correct rating on the ELP performance rubric form;

b. Note here or highlight from the notes above the observed behaviors that substantiate the rating. (Note: this will also be useful during the face-to-face meeting with the student)


Table C: Attitude




Rating Guidance

ATT

ITUDE

1. If the student has a spot evaluation for attitude, is the student’s performance improving?


2. Is student regularly negative or short-tempered with crew or others?

3. Does the student complain regularly and/or blame others?

4. Is the student regularly reluctant to do what’s expected in class, field or lab?

5. Does the student continue to regularly use foul language after being warned?

If the answers to 3 of the 5 questions indicate poor performance and/or an unresolved spot evaluation(s) exists for attitude violations, the student is performing at the Unacceptable level for attitude. Otherwise the student is performing at the Expected level for attitude.

1. Rate the student’s behavior for Attitude using the information documented in the record findings space, the record examples space and the rating guidance information:


b. Note here or highlight from the notes above the observed behaviors that substantiate the rating. (Note: this will also be useful during the face-to-face meeting with the student)


Table C: Housekeeping




Rating Guidance

HOUS

EKEE

PIN

G



2. Does student misplace name tag or doesn’t place on tables in classrooms without prompting

3. Does the crew or others regularly pick up after the student?

If the answers to 2 of the 3 questions indicate poor performance and/or an unresolved spot evaluation(s) exists for housekeeping violations, the student is performing at the Unacceptable level for housekeeping. Otherwise the student is performing at the Expected level for housekeeping.

1. Rate the student’s behavior for Housekeeping using the information documented in the record findings space, the record examples space and the rating guidance information:


b. Note here or highlight from the notes above the observed behaviors that substantiate the rating. (Note: this will also be useful during the face-to-face evaluation meeting with the student).


Table C: Excellence Behavior Criteria Category



Rating Guidance

EXC

ELLE

NC

E 1. If the student has a spot evaluation

for safety, is the student’s performance improving?


2. Is the student always ready to perform competencies or certifications during verification/check-off weeks?

3. Does the student have 3 or more absences or 5 tardies?

If the answers to 2 of the 3 questions indicate poor performance and/or an unresolved spot evaluation(s) exists for attendance/tardy violations, the student is performing at the Unacceptable level for excellence. Otherwise the student is performing at the Expected level for excellence. NOTE: If the student has 3 or more absences or 5 or more tardies AND there is not already a spot evaluation for absence/tardy violations – the student MUST be marked Unacceptable in order to document notification and counsel to student.

1. Rate the student’s behavior for Excellence using the information documented in the record findings space, the record examples space and the rating guidance information:



i.


Table C: Integrity

Continued on page 8




Rating Guidance

INTE

GRI

TY



2. Does student participate in community service projects?

3. Did student cheat, steal, or get caught in a lie?

4. Does student continue to use foul language after being warned?

If the answers to 2 of the 4 questions indicate poor performance and/or an unresolved spot evaluation(s) exists for integrity violations, the student is performing at the Unacceptable level for integrity. Otherwise the student is performing at the Expected level for integrity. NOTE: If the student cheated or was caught stealing – notify the Training Lead immediately to finish the evaluation and documentation.

2. Rate the student’s behavior for Integrity using the information documented in the record findings space, the record examples space and the rating guidance information:




5. If the student is rated Unacceptable in one or more criteria, complete the behavior improvement planner form. a. Circle the behavior criteria rated Unacceptable on the behavior improvement planner form; b. Indicate the improvement due date on the behavior improvement planner form where requested (typically 7 calendar days from the date of the face-to-face evaluation

meeting); c. Note the observed behaviors that substantiate the rating in the comments box on the behavior improvement planner form. d. If there are 3 or more areas rated as Unacceptable on the behavior improvement planner form, consult with the Training Lead to determine next steps.

6. Sign and date the ELP performance rubric form, and the ELP improvement planner if necessary, as indicated on the forms. Note: Once the formal performance evaluations conclude and the students’ have signed their evaluation forms and improvement planners (if applicable), make sure to provide a this completed worksheet along with they signed rubric/planner forms to the Academic Assistant so it’s included in future Student File Packets.


References Chevalier, R. (2003). Updating the behavior engineering model. Performance Improvement, 42(5), 8-14. https://dx.doi.org/10.1002/pfi.4930420504

NLC Instructor: Instructional Design -...

Documents

Transcript of NLC Instructor: Instructional Design -...