Measurable Skill Gains
Nathan CoppleyDirector of Performance
Allison StersicPerformance Assistant
Agenda
• What are the 5 types of MSGs?
➢ What documentation is required for each type of MSG?
• How are MSGs keyed into NCWorks?
• How is performance for MSGs calculated?
➢ Who is included in the denominator?
➢ Who is included in the numerator?
• How is the Period of Participation unique for MSGs?
• Who is excluded from being counted in the measure?
• Where can I learn more about MSGs?
• What is a Measurable Skill Gain?
What to ExpectPolling Questions
• Anonymous
• Helpful to us to make sure you’re getting the information
you need
Question Box• Used to collect your questions at any point during the
presentation
• Questions will be answered and compiled into a
Frequently Asked Questions addendum to the Guidance
Polling Question #1
What is aMeasurableSkill Gain?
Definition of a Measurable Skill Gain
The percentage of participants who, during a program
year, are in an education or training program that
leads to a recognized postsecondary credential or
employment
who are achieving documented academic, technical,
occupational, or other forms of progress, towards
such a credential or employment.
AND
Measurable Skill Gain
The Measurable Skill Gains (MSG) indicator is one of six that make up the performance metrics for the WIOA Title I and Title III programs.
➢ Employment Rate – 2nd Quarter After Exit
➢ Employment Rate – 4th Quarter After Exit
➢ Median Earning – 2nd Quarter After Exit
More specifically, MSG applies
only to WIOA Title I programs:
➢ Credential Attainment
➢ Measurable Skill Gains
➢ Effectiveness in Serving Employers
▪ Adult
▪ Dislocated Worker
▪ Youth
Polling Question #2
What are the 5 types of Measurable Skill Gains?
Types of Measurable Skill Gains
1. Educational Functioning Level (EFL)
2. Secondary School Diploma / Recognized Equivalent
4. Training Milestone/ Progress Report
3. Transcript / Report Card
5. Skills Progression
Educational Functioning Level (EFL)Documented achievement of at least one educational functioning level of a
participant who is receiving instruction below the postsecondary education level.
1. Pre-Test and Post-Test Results
▪ Results from approved tests showing at least one EFL gain
▪ Pre- and Post-tests must be from the same version
2. Enrollment in Postsecondary Education or Training
▪ Participants who exit a program below the postsecondary level and
enroll in postsecondary education or training during the program
yearNote: A program below the postsecondary level applies
to participants enrolled in a basic education program.
EFL is measured in one of the following ways:
Secondary School Diploma or EquivalentDocumented attainment of a secondary school diploma or its recognized
equivalent by obtaining certification of achieving passing scores on all parts of a
State-recognized high school equivalency test.
➢ Copy of credential
➢ Copy of school record
➢ Follow-up survey from program participants
➢ Case notes documenting information obtained from education
or training provider
Accepted documentation includes one of the following:
Transcript / Report Card
▪ Must show grades of D or higher and/or be considered passing
▪ The semester must have occurred within the current program year
▪ The document must reflect that the participant is in good academic standing: nothing to
indicate that the participant dropped out of school or was removed from the institution on
academic/conduct grounds
Postsecondary Education
▪ Transcript demonstrates sufficient number of credit hours:
▪ Full-time – 12 hours for semester within current PY
▪ Part-time – a total of at least 12 hours over the course of
two completed semesters during a 12 month period)
▪ If semester crosses a PY, the MSG would be keyed in
the PY in which the semester ends
Secondary or postsecondary transcript or report card for a sufficient number
of credit hours that shows a participant is meeting the State unit’s academic
standards as follows:
Secondary Education
▪ Documented each
semester.
▪ Specific to youth
attending high school.
Training Milestone / Progress ReportSatisfactory or better progress report towards established milestones from an
employer or training provider who is providing training. Progress reports must
document substantive skill development that the participant has achieved.
➢ Contract and/or evaluation from employer or training provider documenting a skill gain,
or training reports on milestones completed as the individual masters the required job
skills. Increases in pay resulting from newly acquired skills or increased performance
also can be used to document progress if provided as an evaluation from employer.
Acceptable documentation includes:
➢ Documentation of a skill gained (or completed steps) through OJT or Registered
Apprenticeship. Completed steps may be a mid-point evaluation, final evaluation, or
exam results as required by Registered Apprenticeship program.
➢ Progress report from employer documenting a skill gain that the participant has
achieved.
Skills ProgressionSuccessful passage of an exam that is required for a particular occupation or,
progress in attaining technical or occupations skills as evidenced by trade-related
benchmarks, such as a knowledge-based exams.
➢ Results of knowledge-based exam or certification of completion, including a component
exam of a Registered Apprenticeship program
Acceptable documentation includes:
➢ Documentation demonstrating progress in attaining technical or occupational skills
➢ Documentation from training provider or employer such as a satisfactory attainment of an
element on an industry or occupational competency-based assessment
➢ Copy of credential that is required for a particular occupation and only is earned after the
passage of an exam. Examples include: Class A Commercial Driver’s License, Certified
Nursing Assistance License, or CompTIA A+ Certification
Polling Question #3
How are MSGs keyedinto NCWorks?
How is performance for MSGs calculated?
MSG Percent Achieved Calculation
The Measurable Skill Gains indicator is calculated by dividing the
number of participants who attain at least one type of gain during
each period of participation within a given program year (A, below)
by the total participants who are in an education or training program
that leads to a recognized postsecondary credential or employment
(B, below).
MSG
% Achieved
A
B
Denominator Inclusion
B
Participants who, during any point in the program year, are in an education or training program
that leads to a recognized postsecondary credential or employment are included in the
denominator. This number includes participants who do not exit the program and continue to
receive services beyond the end of the program year, as well as those who have exited the
program by the end of the program year.
Title I Adult and Dislocated Worker
• All participants who are in a Title I Adult- or
Dislocated Worker-funded training
program
• Training programs for a secondary school
program equivalent
• Work-based training
Title I Youth
• All ISY (in school youth) are included
• OSY (out of school youth) in the following are
included:
o Occupational skills training
o Secondary education at or above 9th grade level
o Postsecondary education
o Title II-funded adult education at or above the 9th
grade level
o YouthBuild program participants
o Job Corps participants
Title I Adult and Dislocated Worker: All participants who are in a Title I Adult or Dislocated Worker-funded training
program are included in the Measurable Skill Gains indicator (which includes
funding a training program for a secondary school program equivalent). This
includes all participants in work-based training.
Denominator Inclusion Based
on Service/Activity Codes
300 – Occupational Skills Training – on ETPL
301 – OJT Training (Not WIOA Youth)
302 – Entrepreneurial Training
304 – Customized Training
310 – Occupational Skills Training for Special Grants
314 – Registered Apprenticeship
328 – Occupational skills Training – Not on ETPL
333* – TAA – Approved Remedial Training (for those
with GED/HS Diploma)
335* – TAA – Approved Occupational Skills Training
– Approved by State
339* – TAA – Approved GED Training
222* – TAA – English as a Second Language (ESL)
*denotes TAA-only
Service codes that will include Adult and Dislocated Worker in the denominator:
Out School Youth:Only Out of School Youth (OSY) who are in one of the following are included in the
denominator:
407 – Alternative Secondary School services or dropout recovery services
416 – Youth Occupational Skills Training - on ETPL
424 – NC Registered Apprenticeship Training – Youth
429 – Enrolled in Traditional Secondary School (H.S.)
430 – Youth Occupational Skills Training - Not on ETPL
441 – Entrepreneurial Skills Training
Denominator Inclusion Based
on Service/Activity CodesIn School Youth:All In School Youth (ISY) are included in the Measurable Skill Gains indicator since
they are attending secondary or postsecondary school.
Numerator InclusionThe numerator is the number of program participants who achieved at least one type of measurable skill gain.A
1. Educational Functioning Level (EFL)
2. Secondary School Diploma / Recognized Equivalent
3. Transcript / Report Card
4. Training Milestone/ Progress Report
5. Skills Progression
Note:
▪ A participant may have achieved more than one type of gain in a reporting period, but only one gain per participant in a reporting period may be used to calculate success for the measure.
▪ However, all MSGs should still be recorded regardless to ensure success for the participant in the indicator calculation
How is the periodof participationunique for MSG?
Periods of Participation
So what does that mean? Two key factors:
1. Each unique program entry date (not exit date) triggers inclusion
in the calculation.
Unlike the other WIOA performance measures…
MSG is not an exit-based measure!
➢ However, if a participant has 2 periods of participation within 1 PY,
they could appear in the denominator twice and therefore could
receive 2 potential successful MSGs*
*A participant is included in the denominator more than one time during a PY if they exit the
program and are subsequently reenrolled in a WIOA Title I program later in the same program
year AND they participate in an education or training program during each enrollment.
Periods of Participation
2. The MSG indicator is a PY-to-PY measure
➢ One MSG outcome can be achieved in each continuing program year that
a participant is active
➢ If a participant was enrolled in an education or training program during PY
2019 and continued to receive services into PY 2020, that individual could
receive an MSG in PY 2019 AND an MSG in PY 2020.
Periods of Participation
▪ A participant can achieve a Measurable Skill Gain while still participating
in a program
▪ If you forgot to key an MSG at the time…
➢ a Successful Measurable Skill Gain may still be keyed in NCWorks
after the participant has exited the program as long as it is before
the end of the same PY in which they exited. (see FAQ slide)
▪ Key all MSGs
Reminders:
Periods of Participation – An ExampleMSG Opportunity #1
in Pink
▪ Abby’s 1st period of participation only occurs within PY18 and represents MSG Opportunity #1.
MSG Opportunity #2
in Blue
MSG Opportunity #3
in Tan
▪ Later in PY18, she reenrolls for her second period of participation (MSG Opportunity #2).
Explanation:
▪ That period of participation then crosses over into PY19 (MSG Opportunity #3).
▪ This second period of participation results in two inclusions in the denominator because it crossed over
from one Program Year to the next.
▪ Therefore, Abby will be in the MSG denominator two times for PY18 and one time for PY19.
Who is excluded fromthe MSG measure?
Exclusions
a. Institutionalized
Under what circumstances can a participant be excluded
from performance?
b. Health/Medical
c. Deceased
d. Reserve Forces called to Active Duty
e. Foster Care (for Youth participants only)
Where can I learnmore about MSGs?
Additional Resources
▪ Workforce GPS – Measurable Skill Gains E-Module https://tinyurl.com/yxvt9kpj
▪ Interactive Timing Chart by Future Works: A visual tool for understanding the
relationships between the exiting cohort, when they are being measured, and
when they are being reported for each indicator. https://tinyurl.com/y2fqgcrm
▪ TEGL 10-16 Change 1 - Performance Accountability Guidance for Workforce
Innovation and Opportunity Act (WIOA) Title I, Title II, Title III, and Title IV Core
Programs;
▪ TEGL 23-19 – Guidance for Validating Required Performance Data Submitted by
Workforce Programs;
▪ TEGL 7-18 Guidance for Validating Jointly Required Performance Data Submitted
under the Workforce Innovation and Opportunity Act (WIOA)
Attachments to MSG Guidance
1. Guide to Entering MSGs in NCWorks
Staff Online Resources
Attachments to MSG Guidance
2. MSG Desk Reference
Staff Online Resources
FAQ – Submitted so far…Q1: What happens if my participant exited and I forgot to key an MSG?
A1: The MSG can still be keyed as long as it is still in the same program year. For example, if the
participant exited March 30th, 2020 and you realize on June 10th, 2020 that they in fact achieved an MSG
on February 15th, 2020, as long as you have the proper documentation, that MSG can still be keyed to that
participant, even though they are already exited. This is only true within that program year, however. It
can NEVER extend beyond the same program year in which the participant exited.
Q2: Is there a set protocol on what case managers are to do when someone on their caseload falls into
one of the exclusions?
A2: Global exclusions should only be utilized when there are circumstances beyond the control of both the
participant and the program that are expected to last for an undetermined period of time beyond 90 days.
When this occurs, a case note should be entered detailing the circumstance and documentation to support
should be obtained, as applicable.
We urge you to follow any local policies that may be applicable and if you have questions about a
particular circumstance, to coordinate with your superuser and potentially your Regional Analyst, if
necessary.
FAQ – Submitted so far…Q3: I work with out of school youth. When someone passes 1 or 2 of the required 5 HiSET to attain their
High School Equivalency Diploma, can I enter that as an MSG?
Q4: Can we use attaining a Career Readiness Certificate for an MSG? If so, what would be selected for
that since CRC is not specifically listed as an option?
A3: No, it can only be entered as an MSG if all elements of the High School Equivalency have been
completed. In this example, all 5 HiSET component tests would need to be successfully completed to
count as an MSG.
A4: Possibly. If the CRC is a required step in the individual’s training or a normal prerequisite to
employment in a particular industry, then it can be entered as a Skills Progression MSG since the
participant would be passing a test to achieve the CRC.
FAQ – Submitted so far…Q5: If a participant is in a program for more than two program years, can I enter all the MSGs at the end
and have them count toward the first program year as well?
A5: NO!!! All MSGs must be entered within the Program Year in which they occur. We would advise you
to key them as soon as you have the necessary documentation. Waiting could result in you forgetting to
key the MSG and it not counting towards positive performance.
Q6: Can work experience be used as an MSG? Does this vary by program?
Q7: Does OJT count as an MSG and if so, does that apply to all programs?
A6: No, an internship or other work experience is not considered education or training and therefore does
not put the participant in the denominator of the MSG measure. This does not vary by program.
A7: For the Adult and Dislocated Worker programs, OJT counts as an MSG.
However, for Youth, because Youth OJT is considered a work experience and not training, Youth OJT
does not count as an MSG.
FAQ – Submitted so far…
Q8: Will this recorded webinar be available for later
viewing on the Training Center site?
A8: YES! It will be available on the Training Center
website a week or so after the webinar.
Polling Question #4
Thank You
Nathan CoppleyDirector of Performance
Allison StersicPerformance Assistant
Top Related