Glfes summer institute2013_raleigh_final

Welcome!Please …1. Take a seat with your grant team members (or

anywhere if you are solo).2. Grab some coffee.3. Complete the GLF Evaluation Capacity Assessment.4. Get connected to the wireless network: NCSU

Guest. Instructions are in your folder.5. Get access to the GLF Essential Skills Evaluation

“Moodle” website (where we will organize project information). Instructions are in your folder.

Accessing the Moodle Website

Before we get started, we want to make sure everyone has successfully accessed the private Golden LEAF Essential Skills

Evaluation Moodle website.

• Instructions are in your folder.• Raise your hand if you are having

difficulty and an evaluation team member will assist you.

Golden LEAF Essential Skills Initiative

Summer Evaluation Institute

June 27, 2013The Friday Institute – Raleigh, NC

Golden LEAF Essential Skills Initiative Evaluation Team

The Friday Institute at NCSU• Jeni Corn• Latricia Townsend• Rob Maser• Malinda Faber• Laurie Brummit

NCSU College of Education• Diane Chapman• Julia Storberg-Walker

NC Community College System• Kristen Corbell

Agenda

• Welcome and Introduction• The Evaluation Process & Logic Models• Team Logic Modeling• LUNCH• Asking Good Questions• Balanced Scorecard• BREAK• Culture of Curiosity & Data Definitions• Closing and Q&A

The Golden LEAF Essential Skills Initiative

“Supporting collaborative programs aimed at increasing the talent pool of a highly skilled technical workforce aligned with identified employment opportunities in tobacco-dependent, economically distressed and/or

rural communities of North Carolina.”

14 Grants22 Community Colleges

14 School Districts

The Initiative Evaluation

• Grant artifacts• CC registrar data• Student surveys• Instructor surveys• Employer surveys• Focus groups• Interviews• NC Education Research Data Center

administrative data• NC Community College System

administrative data• Employment Securities Commission

workforce data

Evaluation Questions Data Sources

TBD

Who Is in the Room?

One person from each grant team stand up and please share:

•What Community Colleges your team is representing• One sentence description of your

grant’s main focus

The Evaluation Process

Plan, Do, Study, Act (PDSA)

Loyola University Stritch School of Medicine, 2011, http://www.stritch.luc.edu/lumen/MedEd/softchalkhdht/CMEFacDevWebPage/CMEFacDevWebPage10.html

CYCLE OF CONTINUOUS IMPROVEMENT

Five Evaluation Steps

1. Logic Model: Identify the critical elements of the workforce development program.

2. Evaluation Questions: Ask important questions about your workforce development program.

3. Data Sources: Identify what data are available to help answer your questions and determine the additional data needed.

4. Data Analysis: Collect, analyze, and interpret data to answer your questions.

5. Decisions: What changes to your workforce development program should you make based on your result.

Repeat steps 1-511

Communicate & Collaborate

• Include key people in the evaluation planning process:• Especially the primary users of

the results.• This is a great opportunity to

build a shared understanding and buy-in for your project.

• Communicate regularly that this is on-going process of improvement – “This is not a gotcha evaluation.”

Keep it Simple and Focused

• An evaluation doesn’t have to be big! Match the number of evaluation questions in your plan to your resources.

• Focus on efficient, effective data collection strategies.

A basic chart is a great way to organize your own thinking and to share easily the plan with others. Evaluation Questions

Data Sources

STEP 1: Logic Model

Identify the critical elements of the workforce development program

Formative Evaluation Steps

1. Logic Model: Identify the critical elements of the workforce development program.





Repeat steps 1-5

15

ACTIVITYWrite answers to questions 1-3, answering them in that order, on post-its. You will likely have multiple answers - write one idea per post-it.

1. Ultimately, what good is your program going to do (for students, employers, etc.)?

2. If you are successful implementing grant activities, what will your program accomplish this year?

3. What are the major activities of your program? (How are you going to spend your time, money, and other resources?)

Arrange your post-its on a piece(s) of paper from left to right, from major activities (3), to immediate results (2), to the ultimate good the program is going to do (1).

3 2 1 .

What is a logic model?

A logic model is a graphic representation of the

relationships among the key elements of a project: • inputs, • strategies, • objectives, • long-term goals.

A logic model …

• Helps to articulate the key elements of the project.

• Enables evaluation efficiency and effectiveness.

• Promotes stakeholder buy-in by helping clarify how the project works.

• Provides a great opportunity to involve stakeholders in planning.

They are living documents – they can change over time!


www.ScienceCartoonsPlus.com


Inputs Activities Outputs Short-term Outcomes

Medium-term Outcomes

Long-term Outcomes

What is invested in the process? Time, money, human

resources, partners,

equipment, etc.

What are the major

activities the program will entail? What will be done

with the time, money, etc.?

How many participants

are involved? How many teachers and/or

students are reached? What is

produced?

Within life of grant:

What changes in knowledge,

skills, attitudes,

motivations, or awareness

resulted?

Within 1-3 Years:

What changes in behaviors,

practices, policies, or procedures resulted?

After 3 or more Years:

What are the ultimate goals

and did changes results? Student

achievement, teacher

retention, school culture,

etc.

What is a logic model?Terms aren’t consistently used … meaning is what matters.


Whenever possible, it is helpful to try to use the “SMART Goals” strategy for creating your target

outcomes:

SpecificMeasurableAttainableResults-orientedTime-bound

E.g.: Increase number of students taking advanced science courses by 25% by fall 2015.

Strategies Short Term Goals Long Term Goals

Regional career work-based experiences supported by business community

Partnerships to provide enrichment activities for students and differentiated PD for teachers

Curriculum alignment for vertical and cross curriculum through PLCs

Increased access to content specific technology for science and math

Increase enrollment in Algebra 1 for middle school students and upper level math and science

Increase percentage proficiency in Math and Science

Increase use of inquiry-based learning in lessons

Raise STEM interest and awareness

Increase student engagement and achievement in the STEM courses

23

Development of Leadership for supporting STEM instruction

PD in Inquiry-Based learning

Increase student interest and teacher awareness in STEM careers

Promote STEM educator effectiveness through professional development

Early diagnosis and intervention for college readiness assessment

Team Logic Modeling 11:00am – 12:15pm 1. In your grant teams, on a computer, draft a logic

model for your grant. (Use PPT, Word, Inspiration, …) 2. After drafting your logic model, reflect on your PMPs

- are there changes you want to make? 3. (You will get another 20 minutes to reflect on your PMPs in the afternoon.)

Lunch!12:00 – 12:30pm

Pack up your tables a little, grab lunch, and find a new table to sit at

with a conversation topic you’re interested in (look for table tents).

THIS AFTERNOON: Asking Good Questions; Balanced Scorecard; Community College System Data Initiative.

Asking Good Evaluation Questions

Formative Evaluation Steps1. Logic Model: Identify the critical elements of the workforce

development program.





Repeat steps 1-5

27

ACTIVITY

What do you think of when you think of the term “evaluation question”?

Look at your logic model. Jot down 1 or 2 examples of what you think could be evaluation questions for your program.

Developing Evaluation Questions• Evaluation questions provide the direction and

foundation for the entire evaluation.• The process for identifying the questions to be answered

by the evaluation is critical.

EVALUATION QUESTIONS

Data Collection

Data Analysis

Results

Developing Evaluation Questions

Why do we need to ask good questions?

• To determine what is really important, and to whom• Considering “to whom” will help you when you need to

decide what kind of data you want to collect to answer the particular question.

• To focus data collection efforts• What do we need to find out?• How will we collect that information?


Every evaluation question can’t be answered - finding out the answers costs time, money and people. Pick the most important questions that provide the most valuable information to users.

Keep it Simple and Focused

• An evaluation doesn’t have to be big! Match the number of evaluation questions in your plan to your resources.

• This will help you focus on efficient, effective data collection strategies.

A basic chart is a great way to organize your own thinking and to share easily the plan with others.

Evaluation Questions

Data Sources


The main types of evaluation questions are:

1. Questions about STRATEGIES: these questions ask about how well the strategies were implemented.

2. Questions about OBJECTIVES: these questions ask about impacts.

Logic models are great guides for developing evaluation questions.


Quick tips for writing good questions:

1. Try to avoid simple “yes or no” questions2. Consider QUANTITY questions, e.g:

– “How many”– “How much”– “How often”

3. Consider QUALITY questions, e.g.:– “How well”– “How effectively”– “In what ways”

4. Be able to be tuned-in to unexpected results.


IMPLEMENTATION/STRATEGY questions:

- How many hours of sleep am I getting each week? (quantity)

- How soundly am I sleeping? (quality)

IMPACT/OBJECTIVE questions:

- How much weight have I lost? (quantity)

- How has my stress level changed? (quality)

ACTIVITYBrainstorm a list of evaluation questions, and write them down.

1. Brainstorm implementation questions about the program’ strategies

For example:THE STRATEGY IS: Teachers attend 4 quarterly sessions on science kits.A QUANTITY QUESTION: How many faculty/staff attended the sessions? A QUALITY QUESTION: How did the participants rate the quality of the sessions?

2. Brainstorm impact questions to evaluate how well the outcomes are being met.

Crosswalk your list of evaluation questions with your PMP. Do the activities and outcome measures in your PMP capture all of your evaluation questions and the data you would use to answer them? Fill in any gaps.

Balanced Scorecard

What is the Balanced Scorecard?• A 4-quadrant framework that provides a comprehensive

picture of performance• Originally designed for business and industry, now regularly

adapted to different sectors– Original four quadrants are financial, customer, internal processes,

learning and growth

• Adapted to the GLF Essential Skills project, the quadrants are partnerships, workforce education, internal processes, learning and growth

• GLF evaluation objective: Aggregate reporting of four quadrants

Development of the Balanced Scorecard

Combined Evaluation

Strategy

Workforce Education

Internal ProcessPartnerships

Learning & Growth

Combined Evaluation

Strategy

Workforce EducationOutcomes associated with changes

in educational access and achievement including dual degree

and 3rd party certifications

Internal Process

Outcomes associated with changes in enrollment and capacity

PartnershipsOutcomes associated with partners

(community, industry, education) including partnership strength,

activities and impact

Learning & Growth

Outcomes associated with changes in grantee knowledge, skills,

attitudes, and capabilities

Essential Skills Evaluation Balanced Scorecard

Gallery Walk – What are the data sources?

Combined Evaluation

Strategy

Workforce Education

Increase access to programs in targeted areasIncrease program completion in targeted areas attributed to grant proposal and collaborative partnershipsIncreased dual degree opportunities attributed to grant proposal and collaborative partnershipsIncrease in 3rd party credentials attributed to grant proposal and collaborative partnerships

Internal Process

Increase enrollments in targeted courses and programs Increase capacity attributed grant proposalIncrease capacity attributed collaborative relationships

Partnerships

Increase students servedIncrease student employment in targeted industriesStrengthen and increase targeted industry partnershipsIncrease activity with partners

Learning & Growth

Increase learning attributed to collaborative partnerships

(community, students, industry, colleges)

Break!

For 10 minutes.

Hope Opportunity Jobs

Dr. Kristen Corbell, Coordinator of Research ProjectsNorth Carolina Community College System Office

Basic Skills Focus Area Team Chair, Data Initiative Coordinating Team member, Data Access and Advisory Team member, External Sources Focus Area Team Member

Data InitiativeOverview

04/10/2023


Data Initiative Overview

The Data Initiative includes the review, assessment and revision of our current data collection and reporting processes that results in a robust data system which provides accurate and accessible informationfostering a culture of data-driven decision making which addresses research questions and informs policies. 44


Data Initiative Motivation• Concerns over data quality at the college and state level

• Increasing demands for data from federal, state, and grant-providing agencies/organizations

• Expectations of collaboration between state agencies to share data

• The disconnect between data access and decision makers at the college and state level

• Increasing demands on colleges from accreditation bodies to practice and exhibit informed decision-making

• Current reporting tools do not adequately supporting the increasing research and reporting demands

• The need to ensure policies are based on sound data 45


Data Initiative Objectives• Expand the individual and collective inquisitiveness of stakeholders

within the NC Community College System to ask research questions that inform decisions and policies.

• Develop data definitions based on research needs and reporting requirements.

• Ensure data quality and consistency through appropriate and uniform entering and collecting of information across the System so data extracted are valid and reliable.

• Incorporate a data review process that ensures accountability through the validation of submitted data.

46


Data Initiative Objectives• Develop focused topic-based data marts that will serve a wide

variety of analytical research needs including, but not limited to, strategic initiatives, grants, and business and student centric needs.

• Expand information accessibility by making all data users aware of the availability of predefined reports, web-based dashboards, statistical software, and training.

• Enhance advanced analytical capabilities to empower researchers focusing on specific topics and initiatives.

• Utilize research findings and reports to educate policy makers on the extent to which various factors impact outcomes, including student enrollment, student learning, student completion, faculty and staff development, budget allocation, etc. 47


ContinuousImprovement Cycle


Culture of Curiosity / Strategic Initiatives

49

Culture of Curiosity

Represents individual and collective inquisitiveness to ask research questions that inform decisions and policies

Data Needs•Strategic Initiative Variables•Reporting Mandates•Research Needs

Data Definitions

Developed based on research needs and reporting requirements


How the Essential Skills Grant fits in with the Culture of Curiosity

• Within this grant you are asking questions about how the innovation at your college(s) is impacting student success and employment outcomes. These questions are building your culture of curiosity.

• From your questions, you are identifying the data that will be collected to answer the questions.

• For analysis throughout and at the end of the grant, all data must be collected with clear definitions from the beginning and carried throughout.

10/17/2012


Data Quality, Consistency, and Accountability

51

Data Quality and Consistency Ensures information is collected and entered appropriately and uniformly across the system so data extracted for reporting purposes is valid and reliable

Data DefinitionsDirect what, where, when, and how information is entered into the data collection systems

Data Entry Proper understanding of data definitions will sustain the quality of the data collection and reporting processes

Data ReviewProcess which ensures accountability through the validation of data submitted for reporting purposes


How the Essential Skills Grant fits in with Data Quality and Consistency

• Using the data definitions, your college should ensure that the data are collected and entered uniformly.

• We would suggest reviewing the data entered prior to submission to ensure the data make sense. For instance, make sure dates entered are logical, program codes are correct, and student grades are recorded on time for submission with the semester files.

10/17/2012


Data Collection and Warehousing

53

Data Extraction, Transformation, and LoadingProcedural routines of extracting data files from local systems to a centralized warehouse

Data MartsThe creation of specific data sets dedicated to strategic initiatives and research focus areas

Data AccessThe ability to query variables within data marts for basic informational purposes and detailed research needs

Data Collection and WarehousingCentralized and technical processes that impact the way in which data is transferred and transformed between data systems and stored for reporting purposes


Culture of Evidence / Policy Implications

54

Culture of Evidence / Policy Implications•Determination of optimal decisions and policies based on the analysis and reporting of collected data•Likely to result in development of additional research and evaluation needs

General ReportingAutomated informational reports updated routinely

Advanced AnalyticsResearch and analysis focused on specific topics and initiatives to inform current and future practices.

Research Findings & ReportingEducate policy makers on the extent of impact of various factors on outcomes, including student learning and progress

Wrap-Up

Calendar of Evaluation Activities – Year One

ACTIVITY DATESummer Institute 2013 June 2013Grant Coordinator Phone Interview Fall 2013Site Visit Fall-Spring 2013Surveys Winter 2014

Please Take 5 Minutes to Complete Survey

Go to GLF ES Moodle site.

Go to the Summer Institute 2013 Section

Click on the Essential Skills Professional Development Summer 2013 Survey link.

Thank you.

Thank You

Questions? Comments? Concerns?

Have a good trip home!

Michael Maslin Item #: 8476186

Glfes summer institute2013_raleigh_final

Education

Transcript of Glfes summer institute2013_raleigh_final