Evaluator Training Program Introduction · Similarly, Evaluators with a hot tapping background...
Transcript of Evaluator Training Program Introduction · Similarly, Evaluators with a hot tapping background...
1 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
Evaluator Training Program
Introduction
This training will provide Evaluators with the knowledge to perform an evaluation in accordance with their
company’s OQ program.
The steps and processes outlined in this training program are minimum requirements that are common to
pipeline operators who have adopted this program. For specific requirements, please review your company’s OQ
plan, procedures, and practices.
The Evaluator Training Program is a five-part series. The series will cover roles and responsibilities of an Evaluator,
covered task criteria and evaluation methods, span of control, records of evaluations, and evaluator protocol on
how to conduct an evaluation. You should review all modules in the series before attempting the final exam. We
recommend that you download and print a copy of the Evaluator Training Program manual which accompanies
this course. The manual can be accessed in VeriSource and in the resources tab of this course.
Module 1
Upon completion of this module, you will be able to:
1. Describe the definition of qualified.
2. Explain the key terms that make up the definition of qualified.
3. Describe the purpose and the contents of a covered task list.
4. Explain the role and requirements of an Evaluator.
The Department of Transportation’s Operator Qualification Rule requires pipeline operators of DOT-regulated
pipelines to develop a qualification program. The intent is to ensure a qualified workforce in order to reduce the
probability and the impact of accidents/incidents caused by human error.
The Department of Transportation defines the word “Qualified” as an individual who has been successfully
evaluated, can perform assigned covered tasks, and can recognize and react to abnormal operating conditions, or
AOCs, associated with the covered task.
Through evaluations, Evaluators play a major part in determining whether an individual is qualified or not.
Therefore, it’s important for Evaluators to have a good understanding of what it actually means to be qualified.
Before going any further, let’s take a moment to discuss some of the key terms that make up the definition of
qualified.
An individual is a person who, on behalf of the Operator, performs one or more covered tasks on a pipeline facility
operated by the Operator. This includes Operator employees, contractors, sub-contractors, and other entities.
Pipeline operators must ensure that all individuals who perform covered tasks are qualified to do so, or are
working under span of control. Span of control will be discussed in more detail later in the course.
2 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
An evaluation is a process established and documented by the Operator to determine an individual’s ability to
perform a covered task by any of the approved evaluation methods. To evaluate someone means to carefully
assess or judge a person’s ability against an established set of standards or criteria.
Pipeline operators must have evaluation criteria for each covered task. This criteria is then used by the Evaluator to
determine an individual’s qualified status.
A covered task is an activity, identified by the Operator, that:
1. Is performed on a pipeline facility
2. Is an operations or maintenance task
3. Is performed as a requirement of 49 CFR 192 or 195
4. Affects the operation or integrity of the pipeline
Note: If the Operator determines that an activity poses significant risk to the integrity of pipeline facilities or if
an activity is deemed a “covered task” through some other means or reference, the Operator may choose to
make such activity a covered task.
An AOC is “a condition identified by the Pipeline Operator that may indicate a malfunction of a component or
deviation from normal operations that may:
• Indicate a condition exceeding design limits or
• Result in a hazard(s) to persons, property, or the environment.”
A qualified individual must be able to recognize and react to AOCs that are associated with a covered task. The
term “recognize” refers to the ability to identify the AOC based on sight, hearing, smell, and touch. The term
“react” refers to the ability to properly respond to the AOC.
Covered Task Lists – Covered Task Identification
As required by the Operator Qualification Rule, each pipeline operator is responsible for identifying and
documenting their covered tasks. Your company has identified and documented their covered tasks on their
covered task list. The covered task list can be a helpful tool for the Evaluator in preparing for an evaluation as it
contains relevant information about each covered task.
This is an example of a covered task list. Although your company’s covered task list may be different, most
pipeline operators will share the following categories of information.
A typical covered task list will include the task ID, the task description, regulatory reference(s) for the task (if
applicable), the evaluation method for the task, whether the task can be performed by a non-qualified individual
working under the direction and observation of a qualified individual, the span of control limit, and the
requalification interval.
Please keep in mind that your company may include additional categories of information and/or requirements to
their covered task list. Therefore, your covered task list may look different than the sample covered task list
presented in this course.
3 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
Your company covered task list can be found inside VeriSource under the OQ tab. VeriSource is the OQ
compliance management software that is provided by Veriforce.
Authorized Evaluators – The Role of the Evaluator
So far we’ve discussed some of the fundamental aspects of operator qualification. But perhaps the most important
aspect of the entire process is the role of the Evaluator.
An Evaluator is a subject matter expert who is responsible for evaluating whether a Candidate is qualified to
perform a covered task based on your company-specific criteria. As subject matter experts, Evaluators must be
technically competent in the tasks they will be evaluating others on. For instance, Evaluators with a technical
competence in welding would evaluate Candidates on welding covered tasks. Similarly, Evaluators with a hot
tapping background would evaluate Candidates on hot tapping covered tasks, and so on.
Another requirement for Evaluators is that they must be authorized by the pipeline operator in order to conduct
evaluations. In addition, Evaluators must also be re-authorized according to the operator’s plan. Some plans
require an annual re-authorization, while others are on a three year re-authorization schedule. Because pipeline
operators have different requirements for authorizing and re-authorizing Evaluators, it’s important to check with
your OQ Administrator for your specific requirements.
An important concept for Evaluators to understand is the difference between Authorization and Qualification.
Evaluator authorization does not automatically grant qualifications. If the Evaluator decides to pursue qualification
on a given covered task, he/she must be evaluated by another Authorized Evaluator. Some companies require a
triangle method to mitigate evaluation issues, while others require supervisors to perform the evaluation. Please
check with your OQ Administrator for specifics. In addition, some operators may require the Evaluator to be
qualified in the tasks he/she will be evaluating on in order to be authorized as an Evaluator.
So why Evaluators? There are certain reasons why pipeline operators choose to use technically competent
Evaluators in their qualification process. For one, technically competent Evaluators understand the challenges and
obstacles that may occur while the task is being performed. They are able to assess a Candidate’s knowledge,
skills, and abilities, and stop the Candidate prior to performing an incorrect act. Also, if a Candidate is found to be
unsuccessful during an evaluation, a technically competent Evaluator can explain why, whereas a non-technically
competent Evaluator cannot.
Congratulations! You have successfully completed Module 1. To review:
1. The Department of Transportation defines qualified as an individual who has been successfully evaluated,
can perform assigned covered tasks, and can recognize and react to AOCs.
2. A covered task list is a document that is used by pipeline operators to identify and document their
covered tasks. It contains relevant information about each covered task and can be a helpful tool for
Evaluators in preparing for an evaluation.
3. An Evaluator is a subject matter expert who is responsible for evaluating whether a Candidate is qualified
to perform a covered task based on operator-specific criteria.
4. An Evaluator must be technically competent and authorized by the pipeline operator in order to conduct
evaluations.
4 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
Module 2
Upon completion of this module, you will be able to:
1. Identify the most common types of evaluation criteria.
2. Identify the appropriate evaluation method for each type of criteria.
Criteria & Evaluation Methods – Common Types
Criteria may include knowledge points, performance skills, abnormal operating conditions, examinations,
certifications, and any other type determined by your company.
Although the evaluation methods that are used for each will ultimately depend on your company, here are the
most common pairings of evaluation methods and types of criteria.
Knowledge Points: Evaluation Method – Oral Examination
Knowledge points test a Candidate’s knowledge of a covered task. Here is an example of a knowledge point.
Knowledge points have the letter K listed next to them. They consist of a question statement as well as the
appropriate responses. Knowledge points are evaluated through the evaluation method of oral examination. In an
oral examination, the Evaluator questions the Candidate using the knowledge points from the evaluation criteria.
The Evaluator then makes a determination by comparing the Candidate’s response to the response statements
from the evaluation criteria. The Candidate must answer the question correctly, identifying each point that is
measurable.
Note: It’s important to note that the appropriate responses may not be listed on the evaluation criteria. If so,
the Evaluator should defer to the applicable operating procedures in order to assess the Candidate’s response.
When conducting an oral examination, Evaluators should not expect each Candidate to answer the question like
the candidate before him/her or verbatim from the criteria.
In addition, Evaluators may need to “rephrase” or “reword” a question. A best practice is for Evaluators to put
question statements in layman’s terms when Candidates do not understand the question.
It is acceptable to ask additional questions for clarification. However, the questions must be related to the task
and within the standard of the criteria. Do not make the criteria harder or easier than your company’s standard.
You can ask closed or open questions, depending on the situation.
Keep in mind that each Candidate is different. Some will answer each question quickly. Others have a harder time
formulating their thoughts and will need time to think about how to answer.
5 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
A final consideration when conducting an oral examination is to make sure the Candidate can hear you and that
you can hear the Candidate. This is especially important if you decide to conduct the examination outside or in a
noisy environment.
Performance Skills: Evaluation Method – Observation
Evaluation criteria can also be in the form of performance skills. Performance skills test a Candidate’s ability to
perform a covered task. Performance skills have the letter S listed next to them.
They consist of an instruction as well as the appropriate actions or steps. For example, you may see the term
“demonstrate” in the instruction. This should remind the Evaluator that he/she is dealing with a performance skill
and, therefore, the Candidate will need to “perform an action(s).”
Performance skills are evaluated through the evaluation method of observation. During an observation, the
Evaluator watches (or observes) the Candidate perform the skills portion of the task criteria. The Evaluator then
makes a determination by comparing the Candidate’s performance to the actions or steps listed on the evaluation
criteria.
Note: It’s important to note that the appropriate actions or steps may not be listed on the evaluation criteria
for certain performance skills. If so, the Evaluator should defer to the applicable operating procedures in order
to assess the Candidate’s performance.
Observations are perhaps the best way to find out if a Candidate can actually perform a covered task. But just like
in an oral examination, there are certain considerations that an Evaluator should be aware of. When conducting an
observation, Evaluators are actually performing an informed or educated observation. This means that the
Evaluator already knows how the task is supposed to be accomplished. Sometimes there may be more than one
way a task can be performed. However, some tasks require the Candidate to follow a strict step-by-step
procedure. In those instances, there cannot be any deviation from the task steps.
It’s ok to ask the Candidate questions about their performance (either during or after). Sometimes you can obtain
more information if you can get the Candidate to think out loud. A best practice is for the Evaluator to use neutral
prompts. For example:
• What do you see?
• What are you thinking?
• What are you doing?
• What are you looking for?
• What should that be?
Observations may be conducted while someone is actually performing the skill on a regulated pipeline facility (if
the settings are practical). However, as an Evaluator, you should make sure the following considerations are taken
into account.
• Span of control requirements must be addressed, meaning a qualified person may need to be present in
order to direct and observe the Candidate if the task allows it.
• If the Candidate is not successful, the work will have to be verified/reaccomplished by a qualified person.
6 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
• Also, coordination with operations management and Gas Control should be made prior to conducting any
activity.
If the observation cannot be performed on a regulated pipeline facility or if it’s not feasible to do so, then a
simulation can be used. A simulation is the demonstration of a performance skill for the purposes of an
evaluation through the use of a mock setup, rather than a regulated pipeline facility. Simulations should
sufficiently reflect the actual work setting in order to reflect work performance. During a simulation, the candidate
must perform some type of hands-on activity that imitates the actual performance skill to be measured. The
simulation cannot be limited to a candidate talking through the task, using hand gestures, imagining there are
tools and/or equipment present, or describing the activities of other individuals performing the task.
Let’s consider a couple of examples of simulations.
For example, Common Covered Task 008OP Measurement of Wall Thickness with Ultrasonic Device – to conduct a
simulation of this task you could use a scrap piece of pipe, take an actual measurement, and ensure that all steps
are performed as they would be in the actual task performance.
Another example to consider is Common Covered Task 426OP Inspect Pipe Coating with Holiday Detector – to
conduct a simulation of this task you would use a scrap piece of pipe with a working holiday detector in order to
reflect and imitate actual task performance.
A final example is Common Covered Task 412OP Install CP Leads on Pipeline Using Exothermic Welding – during
this simulation you would use a scrap piece of pipe and ensure all materials and equipment are used as they
would be in the actual task performance, performing each step until completion.
Remember, the keys to a proper simulation are to ensure that the simulation reflects the actual work setting and
task performance and to require the Candidate to perform some type of hands-on activity that imitates the actual
skill to be measured.
Should you have further questions about proper simulations, you can access the Veriforce Simulation Guidance
document which can be found in VeriSource under the Resources Tab. In addition, you can also contact the
Veriforce Customer Service group and ask to speak to a Compliance Auditor for further guidance.
AOCs: Evaluation Method – Oral Examination
Evaluation criteria also includes abnormal operating conditions. This criterion tests a Candidate’s ability to
recognize and react to abnormal operating conditions. The criteria consists of a description of the abnormal
operating condition and the proper responses or actions to take in the event the AOC is encountered.
Due to the nature and danger of abnormal operating conditions, an observation or even a simulation is not
practical or recommended to evaluate AOCs. For this reason, AOCs are evaluated through oral examination. When
evaluating a Candidate on AOCs, the Evaluator has the flexibility to address the AOCs in any appropriate manner,
as long as the Candidate has the opportunity to explain how to recognize and react to each AOC.
7 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
One technique is for the Evaluator to describe a hypothetical situation using the examples for the AOC as listed in
the evaluation criteria. The Evaluator then questions the Candidate on the recognition of and the proper
responses or reactions to the AOC. Finally, the Evaluator then makes a determination by comparing the
Candidate’s responses to the evaluation criteria.
Let’s consider an example of this technique in action using the following criteria.
AOC: Unintentional release, vapors, or hazardous atmosphere.
Recognition: Examples include, but are not limited to, blowing gas, bubbles, or dead vegetation.
Response/Reaction: Eliminate potential ignition sources; move to a safe location; notify emergency
personnel; limit access to the location; and follow appropriate procedures.
Given this criteria, the Evaluator may use the following hypothetical situation:
Evaluator – “Let’s say that you were operating this valve, and noticed blowing gas coming from the valve. What
could this indicate?”
Candidate – “An unintentional release or a leak.”
Here the Candidate has recognized the type of AOC based on the hypothetical situation provided by the
Evaluator. But the Candidate must also be able to properly react to an AOC. The Evaluator will need to follow up
with an additional question that tests the Candidate on the react/response portion.
Evaluator – “If it were an unintentional release, what would you do?”
Candidate – “I would eliminate potential ignition sources; move to a safe location; notify emergency personnel, as
appropriate; limit access to the location; and follow the appropriate procedures for notification, documentation,
and remedial action.”
Here the Candidate has described the proper reactions he/she would take to address the AOC. In this example,
the Candidate has now recognized the AOC and has described how he/she would react to it.
Another technique involves the Evaluator asking the Candidate how a particular AOC can be recognized and
asking how they would respond. Using the same sample criteria from the previous example, this is how this would
unfold.
Evaluator – “How could you tell if an unintentional release or leak was happening?”
Candidate – “If there’s evidence of blowing gas, puddles, or dead vegetation.”
Evaluator – “How would you respond?”
8 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
Candidate – “I would eliminate potential ignition sources; move to a safe location; notify emergency personnel, as
appropriate; limit access to the location; and follow the appropriate procedures for notification, documentation,
and remedial action.”
Regardless of what technique is used, Evaluators must ensure the Candidate possesses the knowledge to
recognize and react to the AOCs listed on the evaluation criteria.
Examinations and Certifications: Evaluation Method – Exams and Review
Criteria can also be in the form of examinations. Examinations test a Candidate’s knowledge of a covered task. The
Candidate is either given a paper test or is required to take the exam online, either in VeriSource, your company’s
intranet, or in an another location as specified by your company. Exams are typically completed in conjunction
with computer-based training, where the Candidate must take an e-Learning module and complete the required
exam, although this isn’t always the case. Check with your OQ Administrator for specifics.
Criteria can also be in the form of certifications. A Candidate may be required to provide current certification from
industry organizations, manufacturers, or vendors. Such requirements will be listed in the evaluation criteria of the
covered task (if applicable). An Evaluator should review the Certificate to ensure it is current and that it is the type
required by the covered task.
Congratulations! You have successfully completed Module 2. To review:
1. The most common types of evaluation criteria are knowledge points, performance skills, AOCs,
examinations, and certifications.
2. An oral examination is used for knowledge points
3. An observation (which can include simulation) is used for performance skills.
4. Simulations should reflect actual work setting; the Candidate must perform a hands-on activity; and the
simulation cannot be limited to talking, imagining, or describing activities.
5. AOCs are to be evaluated through oral examination.
6. Examinations are designed to test the candidate’s knowledge of the covered task.
Module 3
Upon completion of this module, you will be able to:
1. Define span of control.
2. Describe how to comply with span of control requirements during an evaluation.
Span of Control: Requirements
The OQ Rule allows nonqualified individuals to perform certain covered tasks on regulated pipeline facilities if
directed and observed by a qualified individual. Span of control is the maximum number of nonqualified
individuals that a Qualified Individual can direct and observe for the conditions under which the task is being
performed.
9 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
While directing nonqualified individuals, the qualified individual must be able to take control of the task; be in
communication with the nonqualified individual, have close supervision over the nonqualified individuals at all
times, and respond appropriately to abnormal operating conditions should they arise.
If the qualified person walks away or for some reason is no longer in control of the task, the nonqualified
individual(s) must stop performing the task.
The span of control limit of a covered task, which is determined by your company, outlines the maximum number
of nonqualified individuals that a qualified person can direct and observe for the conditions under which the task
is being performed. Let’s look at a couple of examples of span of control limits.
Span of Control – 1:3
This ratio indicates that one qualified individual may direct and observe no more than three nonqualified
individuals during the performance of the covered task.
Span of Control – 1:1
A ratio of 1:1 indicates that one qualified individual may direct and observe no more than one nonqualified
individual during the performance of the covered task.
Span of Control – 1:0
A ratio of 1:0 indicates that only qualified personnel may perform the task and that performance of the task by a
nonqualified person is not allowed.
Span of control limits can be found on the covered task list and on the Record of Evaluation. Records of
Evaluation (ROEs) will be discussed in the next module.
Span of Control and Evaluations: Evaluator Options
Span of control (SOC) limits must be enforced anytime an individual performs a covered task on a regulated
pipeline facility whether it be during an evaluation, training, or actual task performance.
Therefore, in cases where actual task performance on a regulated pipeline facility is used as a means to evaluate
the Candidate, the Evaluator must ensure that appropriate span of control limits are followed. There are certain
scenarios that an Evaluator must be able to recognize and address in order to adhere to span of control
requirements. Let’s take a look at each scenario to get a good understanding.
Scenario #1 – The Evaluator is qualified and the Candidate is not qualified.
In this scenario, the Evaluator is qualified on a covered task while conducting an evaluation on a regulated
pipeline facility, so the nonqualified Candidate can perform the task under the Evaluator’s SOC.
Scenario #2 – The Evaluator is not qualified, and the Candidate is not qualified, but a qualified individual is
available to observe.
10 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
In this scenario, the Evaluator could bring in a qualified person to oversee the nonqualified individual’s task
performance on a regulated pipeline facility. However, if the qualified person has to direct or stop the
nonqualified individual’s performance, the evaluation would be considered unsuccessful.
Scenario #3 – The Evaluator is not qualified and the Candidate is qualified.
This scenario can occur if the Candidate is qualified (such as when he’s up for requalification), the evaluation could
take place on the regulated pipeline facility, regardless of the Evaluator’s qualification status or span of control
limit.
Scenario #4 – The Evaluator is not qualified and the Candidate is not qualified.
If neither the Evaluator nor Candidate is qualified and there is no other qualified person around to observe, the
evaluation must be simulated.
Scenario #5 – The Evaluator is not qualified, the Candidate is not qualified, and the span of control for the task is
1:0.
For the last scenario, the evaluation must also be simulated if the span of control for the task is 1:0 and both the
Evaluator and Candidate are nonqualified. Even if there were qualified individuals available and willing to direct
and observe, the evaluation must be simulated since the span of control limit of 1:0 only allows qualified
personnel to perform the task.
Congratulations! You have successfully completed Module 3. To review:
1. The span of control limit of a covered task, which is determined by your company (the operator), outlines
the maximum number of nonqualified individuals that a qualified person can direct and observe for the
conditions under which the task is being performed. An oral examination is used for knowledge points.
2. Span of control (SOC) limits must be enforced anytime an individual performs a covered task on a
regulated pipeline facility, whether it be during an evaluation, training, or actual task performance.
Therefore, in cases where actual task performance on a regulated pipeline facility is used as a means to
evaluate the Candidate, the Evaluator must ensure that appropriate span of control limits are followed in
order to be in compliance with DOT regulations. Simulations should reflect actual work setting; the
Candidate must perform a hands-on activity; and the simulation cannot be limited to talking, imagining,
or describing activities.
Module 4
Upon completion of this module, you will be able to:
1. Describe the use of the record of evaluation (ROE) form.
2. Identify each section of an ROE.
3. Describe how to submit a qualification within VeriSource.
4. Describe how to submit documentation to Veriforce.
Record of Evaluation: Qualification Compliance Document
11 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
Evaluators must use a record of evaluation (ROE) each time an evaluation is conducted. The ROE contains the
evaluation criteria and serves as the qualification compliance document.
ROEs are the first option under the Internal Evaluators Tab in VeriSource.
Once an Evaluator is authorized, he/she will have access to the ROEs they have been authorized on.
Prior to each evaluation, the Evaluator will need to print the ROE and take it with him/her to the evaluation.
Record of Evaluation: ROE Form
This section will introduce a typical ROE in sections.
The very top of an ROE begins with the record of evaluation instructions. This section outlines reminders on how
to conduct an evaluation and how to fill out the ROE. Make sure to fill out the ROE completely, but do not post
comments or write on the first page of the ROE other than where it is required. Never use white-out. If you make a
mistake, cross out the mistake and then initial the change.
The training requirements section requires the Evaluator to ask the Candidate each question that is listed.
• Is the Candidate currently disqualified due to his/her role in an incident/accident or for cause?
• Was the Candidate unsuccessful when last evaluated for this covered task?
• Has a High Impact change taken place related to this covered task since the Candidate was last
evaluated/qualified?
If a change that impacts evaluation criteria, task descriptions, or program requirements occurs, Veriforce and the
applicable pipeline operator(s) will categorize the change(s) as either low, medium, or high impact.
• A Low Impact change may include grammatical, formatting, or other modifications that result in no
material effect on the administration and implementation of the OQ Program.
• A Medium Impact change may include revisions to administrative procedures, evaluation methods,
operating procedures, or other items that affect the implementation and administration of the OQ
program.
• High Impact changes are those that require the requalification of individuals previously qualified on the
applicable task(s). A Candidate whose qualification is revoked as a result of a “High Impact” change is
required to complete training before being re-evaluated.
Note: Veriforce will communicate medium and high impact changes through any or all of the following: by
posting it in VeriSource or by broadcast email, fax, or US Postal Service (at pipeline Operator’s discretion).
So back to the ROE, if the answer is YES to any of the questions, the Candidate must have received training prior
to the evaluation. The Evaluator will need documentation of the training before the evaluation can begin.
(Documentation of training requirements will be discussed later on in the training.) If all of the answers are NO,
there is no need to submit training documentation.
12 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
Note: Your company may have additional training requirement questions listed under this section. Check with
your OQ Administrator for specific requirements.
The employer information section requires the Candidate’s company name. The Candidate information section
requires the Candidate’s first, middle, and last name, and either their date of birth or their Veriforce ID. If they
don’t know their Veriforce ID number, you can fill it in when you submit the qualification data. (Qualification data
is submitted upon the conclusion of the evaluation.) It is extremely important for the Candidate’s information to
be documented exactly as it appears on the Candidate’s government-issued photo I.D. Furthermore, the name
and ID or date of birth must be legible. This information is often prepopulated by the system. However, if
Evaluators need to print this information, it must be legible.
The next section identifies the covered task number and description. This is also prepopulated by the system. The
supporting documentation area will indicate if a certificate or other documentation is required in addition to the
evaluation (e.g., Clock Spring certificate). Under the Evaluator information section, the Evaluator’s name will be
listed. Do not use an ROE with another Evaluator’s name on it. The only way to obtain an ROE with someone else’s
name on it is to log into VeriSource using their user name and password. It is not permissible to cross out the
other Evaluator’s name and insert your own. Evaluators must use their own VeriSource user name and password to
obtain the ROEs they are authorized on.
The results section requires the Evaluator to document the outcome of the evaluation. Check either Successful or
Unsuccessful.
Under Candidate and Evaluator acknowledgement, the Candidate and Evaluator must sign and date the form. The
signatures must be present and dates must be legible.
• Only the Candidate may sign for the Candidate. Only the Evaluator may sign for the Evaluator.
• The dates indicate the date the evaluation criteria was met and the evaluation was completed. This is not
necessarily the submittal date. The Candidate and Evaluator dates must be the same.
The disclaimer and no warranty section explains that the ROE alone does not constitute a qualification – only
documentation that the evaluation occurred. Qualifications are only issued after the ROE and data are submitted
to Veriforce and verified through the quality assurance process. It also explains that the qualification is not valid
for non-Veriforce operators and represents a breach of contract if used for that purpose.
The very bottom of page 1 displays the revision and date of the ROE. The most current revisions are kept in
VeriSource. Be sure to use the most current revision while performing an evaluation. If you submit an outdated
ROE, it will be rejected.
The top of page 2 contains the evaluation method, regulatory reference (if applicable), subsequent qualification
interval, and span of control.
The next area outlines the criteria used during the evaluation and the abnormal operating conditions associated
with the covered task.
13 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
Submitting Qualification Data: VeriSource
Once the evaluation has concluded, the Evaluator must submit the evaluation data in VeriSource. Depending on
your company, you may or may not have to fax or email the first page of the ROE to Veriforce. Some operators
require the ROE to be submitted to their OQ admin for review and submittal. If you are required to submit the
ROE, make sure it is filled out completely and that it is legible.
Note: The steps outlined in this section may be slightly different than your company’s specific process. For
detailed instructions on submitting qualification data, please contact your OQ Administrator.
To submit an ROE:
• In VeriSource, under the Internal Evaluators tab, click Online ROE.
• Next, click on I want to Submit an Evaluation.
• Next, select the Candidate’s name from the drop down list.
• Select the covered task you evaluated the Candidate on. Multiple tasks can be selected by holding the
down arrow and CTRL key.
• Review your contact information.
• Select the date the evaluation was conducted.
• Adjust whether the Candidate was successful or unsuccessful as well as the answers to the training
requirement questions.
• After reviewing the information and making the appropriate selections, submit the evaluation result.
Submitting Documentation: QSF, ROE, and Supporting Documents
This section only applies if you are required by your company to submit the ROE and supporting documentation
to Veriforce. If you are, before faxing or emailing page one of the ROE, you will need to print out the Qualification
Submittal Form (QSF).
When submitting the paperwork, submit the documents in this order:
1. Qualification Submittal Form
2. Page 1 of the ROE
3. Any other supporting documentation (e.g., training record)
Send these documents to 1.866.447.9104 if faxing, or to [email protected] if scanning and emailing.
Before submitting the ROE to Veriforce, double check that all of the required information is filled out on the ROE,
make sure it is legible, and verify the evaluation dates are the same.
If anything is left blank or illegible, the ROE will be rejected. If an ROE submission is rejected, the correction
should be sent to [email protected] with “roefix” in the subject line of the email.
Congratulations! You have successfully completed Module 4. To review:
1. The ROE contains the evaluation criteria and serves as the qualification compliance document. Evaluators
must use a record of evaluation (ROE) each time an evaluation is conducted.
14 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
2. Before submitting the ROE to Veriforce, double check that all of the required information is filled out on
the ROE, make sure it is legible, and verify the evaluation dates are the same. If anything is left blank or
illegible, the ROE will be rejected.
3. The evaluation documentation should be submitted in this order: 1-Qualification Submittal Form; 2-Page
1 of the ROE; 3-Any other supporting documentation (e.g., training record).
Module 5
Upon completion of this module, you will be able to:
1. Identify Evaluator responsibilities prior to an evaluation.
2. Describe how to brief a Candidate.
3. Describe the rules for conducting an evaluation.
4. Describe additional requirements for conducting an evaluation.
5. Describe how to debrief a Candidate.
6. Explain the training requirement after an unsuccessful evaluation.
Preparing for an Evaluation: Pre-Evaluation Activities
Evaluators should make sure that everything needed for the evaluation is available and present. You should
prepare for an evaluation by reviewing the materials that will be used during the session. These include the:
• Evaluation criteria
• Equipment and facilities
• Personnel
• Special needs
Let’s discuss each of these in more detail.
The first item on the list is the evaluation criteria. Each Candidate should review the evaluation criteria prior to the
evaluation. This allows the Candidate time to practice or ask questions. The Evaluator should confirm that the
Candidate has reviewed the evaluation criteria and/or provide it if necessary prior to the evaluation. Under no
circumstances should the Candidate be allowed to use the criteria during the evaluation.
The evaluation criteria can be accessed in two different locations within VeriSource. The first location is under the
OQ Tab. The other location is in the Internal Evaluators Tab under Online ROE. When an ROE is downloaded, the
evaluation criteria will automatically download with it.
As an alternative to the evaluation criteria, a Candidate can review a training guide. Training guides provide an in-
depth explanation of each item of the evaluation criteria.
Note: Please check with your OQ Administrator to determine if your company has training guides available for
their covered tasks.
In addition to the criteria, the Evaluator should also ensure that the necessary equipment and facilities needed to
conduct the evaluation are available. It is your responsibility to make sure the Candidate has access to everything
15 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
needed to perform the task. A best practice is for the Evaluator to make every effort to provide the tools and
equipment that are familiar to the Candidate during the evaluation. Regarding the setting, it is highly
recommended to find a location for the evaluation away from distractions. Candidates will feel more comfortable
and perform to their fullest potential if they feel at ease with their equipment and surroundings.
Personnel is another consideration that an Evaluator must take into account when preparing for an evaluation. As
discussed in a previous module, if the task is to be performed on a regulated pipeline facility for the purposes of
an evaluation, the Evaluator must make sure to comply with the applicable span of control requirements. The
Evaluator may be required to have a qualified person present during the evaluation, depending on the task and
the qualification status of the Evaluator and the Candidate.
If the Candidate requires special needs to perform the task, you should seek guidance from your OQ admin
and/or refer to your company’s policies and procedures.
Briefing the Candidate: Starting the Session
Briefing the Candidate is extremely important for conducting a proper evaluation. It sets the stage for the
evaluation, allows the Evaluator to collect required information, and outlines Candidate and Evaluator
expectations. Briefing a Candidate consists of the following steps:
1. Introduce yourself
2. Break the ice
3. Explain the purpose
4. Provide an overview
5. Confirm identity
6. Determine training requirements
7. Verify supporting documentation
8. Explain the rules
As the Evaluator, you should greet the Candidate with a handshake when the evaluation begins. A handshake,
even to someone you know well, sets the stage for the beginning of the evaluation. The Evaluator should also
introduce him or herself.
Since most Candidates will be nervous, a best practice is for the Evaluator to take a few seconds to calm the
Candidate down by talking about something that interests the Candidate; for example, hunting, fishing, sports,
etc.
To help with Candidate’s anxiety, it’s also a best practice for the Evaluator to explain the purpose of the evaluation.
Often Candidates don’t know why they’re being evaluated and why they are being tested. Explaining the purpose
to the Candidate can help him/her obtain the big picture and can help with any anxiety or nervousness he/she
may have.
Another best practice is to explain how you, the Evaluator, are going to conduct the evaluation; for example, if it’s
a launching and receiving task, you might tell the Candidate that you are going to ask him/her a series of
16 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
questions then go out in the field to watch him/her perform a launching and receiving operation, followed by
questions on abnormal operating conditions.
Once all of the preliminary and background information has been covered, Evaluators should proceed and ask the
Candidate for his/her government-issued photo I.D. This will be used to document the Candidate’s name and date
of birth on the ROE. As mentioned earlier, the Candidate’s name and date of birth should be documented as it
appears on their government-issued photo I.D. The Candidate’s company name should also be documented.
Note: Your operator may have different requirements or methods for confirming the Candidate’s identify (e.g.
the use of a company ID or prepopulated ROE or visual identification of an individual you know). Check with
your OQ Administrator for specifics.
The next step is to determine if training is required prior to the evaluation. To do this, you will need to ask the
Candidate the Training Requirement questions from the ROE. If any of the questions are answered YES, training
must have occurred prior to the evaluation and the Candidate must show proof by providing you with training
documentation.
Note: For some companies, Candidates must complete required training before the evaluation. Otherwise, the
Evaluator will not be able to download the ROE. In this type of system configuration, Candidates will not be
able to provide training documentation since the system tracks this automatically. Check with your OQ Admin
or company procedures to verify training acceptance and additional requirements.
You will also need to verify if the covered task requires any supporting documentation, such as a manufacturer’s
certificate, visual acuity test, etc. To verify, look at the Supporting Documentation field on the ROE. If supporting
documentation is required, the Candidate should provide the necessary documents.
The last step in briefing the Candidate is to explain the rules of the evaluation. Inform the Candidate that:
• There is no time limit – Candidates are allowed as much time as they need to perform the task. However,
it is the Evaluator’s call if they are taking too long.
• You cannot assist with any aspect of the covered task – Evaluators must maintain the integrity of the
evaluation process and therefore should not assist or help the Candidate.
• You will stop the evaluation for any safety violation (such as danger to personnel or equipment) and deem
the evaluation unsuccessful – Candidates must observe all safety rules and must be able to manipulate
equipment adequately.
• You will review his/her performance at the conclusion of the evaluation session. Evaluators should review
the Candidate’s performance at the end of the evaluation so that the Candidate is aware of the outcome.
To finish briefing the Candidate, ask if he/she has any questions before beginning the evaluation.
Conducting the Evaluation: Evaluator Protocol
There are five important rules that an Evaluator must follow during an evaluation.
1. Evaluate the Candidate individually.
2. Ensure the evaluation is done in a one-on-one setting.
17 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
3. Follow the evaluation method as required by the covered task.
4. Adhere to the evaluation criteria of the covered task.
5. Do not train the Candidate during the evaluation.
It is extremely important for you, as the Evaluator, to follow these rules in order to maintain the integrity of the
evaluation process. Let’s discuss each of these rules in more detail.
The first rule requires each Candidate to be evaluated individually. Candidates cannot be evaluated two at a time,
in a group, or over the phone.
The second rule is that each evaluation must be conducted in a one-on-one setting. One-on-one means a
conversation or interaction between two people where no one else is present. One-on-one settings may include
areas such as in an office, vehicle, shop, field, or on the right-of-way.
Sometimes additional individuals may have to be present, such as when a qualified person needs to be present to
comply with span of control limits, when the operator requires internal auditing supervisors to be present, or
when state and federal auditors wish to see an evaluation process.
Regardless of the reason, if others have to be present, they need to meet the following requirements:
• They cannot be future Candidates – future Candidates will mimic what they just heard or saw.
• They cannot make your Candidate feel uncomfortable – no one likes to be tested and having other people
around may make the Candidate uneasy.
• They cannot distract you or the Candidate.
The third rule is that the Evaluator must follow the evaluation method required by the covered task. The required
evaluation method will be listed on the evaluation criteria of the covered task and on your company’s covered
task list. An important thing to note is that if a covered task requires more than one method, both methods must
be followed. For example, for common covered task 007OP operate valves, the evaluation method states
observation and oral exam. In this example, the Evaluator would have to orally examine the Candidate on the
knowledge points and AOCs AND observe the Candidate perform the performance skills of the covered task.
The fourth rule involves the evaluation criteria. This rule requires the Evaluator to adhere to the evaluation criteria
of the covered task. The criteria may not be altered or changed. Your company has established the criteria as task
standards. If for some reason you find that the criteria does not meet your company-specific requirements, get
with your company program administrator so a review can be conducted. Under no circumstances may the
Evaluator deviate from the criteria. Evaluators cannot add to the criteria, are not allowed to make the criteria
harder than what it is, and cannot require the Candidate to know or perform more than what is required by your
company. On the other hand, Evaluators cannot remove from the criteria. Candidates are required to answer each
knowledge point, perform each performance skill, and recognize and react to each AOC correctly.
The last rule prohibits Evaluators from training a Candidate during an evaluation. Training includes any type of
assistance with the task. Evaluators must not suggest a task performance technique or hand out or point out the
location of tools or equipment.
18 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
There will be occasions when a Candidate asks a question and the Evaluator must “relay” the question back to the
Candidate because answering would be considered training. Some examples of how to “relay” these types of
questions back to the Candidate may include:
• Candidate: “Do you think I should clean this pipe better before fusing it?”
• Evaluator: “How would we know if the pipe was cleaned sufficiently?”
• Candidate: “What does this symbol mean?”
• Evaluator: “What do you think it means?”
• Candidate: “Should I check the pressure now?
• Evaluator: “How do we know when the pressure should be checked?”
Be careful not to ask leading questions. A leading question gives the Candidate a partial answer, which he/she is
expected to complete, or gives a complete answer and expects the Candidate to agree or disagree with it. Leading
questions are those that nudge, push, or shove the other person towards a particular answer and away from other
answers. Leading questions are another form of training. For example:
• Evaluator: “Are you going to refer to the procedures before starting the equipment?”
• Evaluator: “Are you going to lockout the machine before opening it?”
• Evaluator: “You are going to don your hardhat and glasses before performing that task – right?”
Additional Requirements: Evaluator Protocol
Following the rules mentioned in the previous section is essential in order to maintain the integrity of the
evaluation process. However, an Evaluator can improve the quality of his evaluations and to some extent can put
the Candidate at ease by adhering to the following items.
Perhaps the number one distraction is the use of cell phones. You should turn off your cell phone and give your
full attention to what the Candidate is saying or doing. Try not to be distracted by your surroundings or your own
thoughts. Candidates will sense that their performance is not important if they believe you are distracted. The
worst thing an Evaluator can do is to undermine the importance of the evaluation.
Pay close attention to the Candidate’s responses or performance. Make sure you don’t try to anticipate what they
are going to say or do, and don’t let your mind wander. Always avoid being in a position where you’re not sure
whether to deem the Candidate successful or unsuccessful because you weren’t paying attention. This could end
up being unfair to the Candidate, or worse, you could end up qualifying someone that really shouldn’t be.
Another item to adhere to is to remain as neutral as possible. If the Candidate picks up in your body language,
facial expressions, or non-verbal sounds (such as sighing) that he/she answered a question or performed a step
incorrectly, it’s possible that they may be unable to continue.
Evaluators should be aware of their surroundings at all times. You should immediately suspend the operation of
equipment during unanticipated or abnormal events, during accident conditions, or whenever you feel it is
necessary to ensure safe and reliable operation.
19 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
Debriefing the Candidate: Concluding the Session
To be successful, the Candidate must have met 100% of the criteria during the evaluation. Once the evaluation is
over, you must review the Candidate's performance. Let him or her know if they were successful or not.
If the Candidate was successful:
• Review the Candidate’s performance with the Candidate.
• Inform the Candidate of the outcome.
• Congratulate the Candidate.
• Complete the ROE – mark the Candidate successful – both the Evaluator and the Candidate must also sign
and date the ROE.
• Inform the Candidate of the span of control limits of the covered task.
• Conclude the session.
• Submit the evaluation data in VeriSource.
• Fax or email the evaluation documentation to Veriforce, as applicable.
If the Candidate was unsuccessful:
• Review the Candidate’s performance with the Candidate
• Inform the Candidate of the outcome
• Explain that in order to be re-evaluated, he/she must fulfill the training requirement.
• Complete the ROE – mark the Candidate unsuccessful – both the Evaluator and the Candidate must also
sign and date the ROE
• Inform the Candidate of the span of control limits of the covered task. If the task’s span of control allows
it, let the Candidate know that he/she is only allowed to perform the covered task if directed and
observed by a qualified person.
• Conclude the session.
• Submit the evaluation data in VeriSource – It’s important to submit unsuccessful evaluations and
documentation. Just because the result of an evaluation is unsuccessful doesn’t mean that Evaluators
don’t have to submit the data and necessary paperwork. Submitting unsuccessful evaluations can help
validate the effectiveness of the Operator’s qualification program.
• Fax or email the evaluation documentation to Veriforce, as applicable.
Notice that the steps for debriefing a successful and unsuccessful Candidate are similar with the exception of the
training requirement. We’ll discuss the training requirement in more detail in the next section.
Training Requirement: Failing Candidates
After an unsuccessful evaluation, Candidates are required to complete training related to the task before they can
be evaluated again.
Note: Some operators may have time restrictions before a Candidate can be re-evaluated or possibly a limit to
unsuccessful evaluations. Check with your OQ Admin or company procedures for specific requirements
regarding the re-evaluation of unsuccessful Candidates.
20 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
As far as the training requirement, the training may be in the classroom, online, or on-the-job (OJT). The training
must be specific to the task the Candidate failed and the training must be documented.
In order for training documentation to be acceptable, it must clearly identify the:
• Name or title of the training program
• Source of training
• Candidate’s name
• Date of completion
Examples of training documentation can consist of sign-in sheets, certificate of completion, attendance sheet, or a
valid record of training (ROT).
An ROT can be used to document on-the-job training. ROTs are available in VeriSource for each covered task. If
the operator allows it, the Evaluator is authorized to conduct training. The Evaluator can train a Candidate before
or after, but NOT during, an evaluation. If you do conduct training as an Evaluator, do not count a good training
or practice session as an evaluation. An evaluation must be a planned, separate event.
During an evaluation, if the Candidate stumbles through the task and begins to fail (non-safety related), the
Evaluator can choose to stop the evaluation or allow the Candidate to finish. Either option is acceptable.
Allowing the Candidate to finish will provide a complete picture of what the Candidate knows and doesn’t know.
This is helpful information with regard to the training that is required after an unsuccessful evaluation. However,
stopping the evaluation may be the best option if it becomes obvious that the Candidate needs more training or
experience. Stopping an evaluation should be handled delicately. Try not to embarrass or demean the Candidate.
Note: Check with your OQ Administrator for more information on Evaluators and their role in the training of
personnel.
As a final point, failing someone is generally difficult to do. It seems to be in our nature to help others when they
are in trouble, or make excuses for their poor performance, especially if it’s a fellow employee that you have
worked with for years or even a family member. However, if someone is failing, do not assist or pass them thinking
they are just having a bad day or that they will improve their skills with time. Doing this violates regulatory
requirements and jeopardizes the integrity of the pipeline and the safety of that individual, everyone around, and
the general public.
Congratulations! You have successfully completed Module 5. To review:
1. The Evaluator must ensure that everything needed for the evaluation is available.
2. 1-Introduce yourself; 2-Break the ice; 3-Explain the purpose; 4-Provide an overview, 5-Confirm the identity
of the Candidate, 6-Determine training requirements, 7-Verify supporting documentation, and 8-Explain
the rules.
3. 1-Evaluate the Candidate individually, 2-Perform the evaluation in a one-on-one setting, 3-Follow the
evaluation method, 4-Adhere to the evaluation criteria, and 5-Do not train during the evaluation.
4. Avoid distractions, pay attention to what the Candidate is saying or doing, remain neutral, and be safe.
21 | ©2017 Veriforce, LLC. All rights reserved.
Rev 10 – 9/19/2017
5. 1-Review performance, 2-Inform the outcome, 3-Congratulate Candidate if successful OR inform of
training requirement if unsuccessful, 4-Complete the ROE, 5-Inform Candidate of SOC, 6-Submit
evaluation data, 7-Submit evaluation documentation, and 8-Conclude evaluation.
6. Complete training related to the covered task.
This concludes the Operator Evaluator Training Program. Please review all modules in the series before attempting
the final exam.
If you have any questions concerning this module, please don’t hesitate to contact Veriforce at 800-426-1604 or
email us at [email protected].