Outcomes-Based Grading: A Direct Measure of Instructional ...
Advice on Data Used to Measure Outcomes Friday 20 th March 2009.
-
Upload
audrey-stone -
Category
Documents
-
view
215 -
download
0
Transcript of Advice on Data Used to Measure Outcomes Friday 20 th March 2009.
Advice on Data Used to Measure Outcomes
Friday 20th March 2009
Today’s Workshop• Introduce myself & background• 2008/09 SOA – SBC experience• 2009/10 SOA – Context• How we developed the 2009/10 SOA
– Consultation • Linking activities to SOA – the 4 areas PED leads• How to measure success
– The outcome– What SBC needs to measure – a worked example
• Issues in what to measure– Primary
• Some conclusions • Discussion
Introduction• Planning & Economic Development
• Business Information Unit has led development of Outcome Agreements
• Own experience of developing evaluation frameworks contributing to PSA in England Local Authorities
• Process of Logical Frameworks, Logic Chains etc
Area of activity• Regeneration – town centre, retail, events &
tourism expertise• Property – economic development, infrastructure
and property management expertise• Economic Development – tourism, inward
investment, business support expertise• Rural Development – economic development,
key sectors, funding, regeneration expertise• Business Gateway – key sectors, business
support expertise
Our Experience
• Scottish Government’s National Outcomes• Development for 2008/09• Scottish Borders Council - Observations
– New way of working– Community Planning Partnership = 20 partners– c 48 Outcomes– c 15+ under ‘economic’– Mix of outcome, indicators, output, activity– Difficulties in performance management
• Different times
2009/10 SOA – Context • NO - We realise our full economic potential with more
and better employment opportunities for our people
• NO - We are better educated, more skilled and more successful, renowned for our research and innovation
• NO - Our young people are successful learners, confident individuals, effective contributors and responsible citizens
• NO - We live in well-designed, sustainable places where we are able to access the amenities and services we need
9 Outcomes that relate to the ‘economy’
• 2.1 – supporting & growing business activity in key sectors
• 2.2 – promoting self-employment & creating sustainable businesses
• 2.3 – maximising employment opportunities
• 10.3 – improving the socio-economic performance of our town centres & growing stronger communities
• 3.1 – maximising participation in education, training & employment amongst economically inactive
• 3.2 – existing workforce is highly skilled and responsive to the needs of employers
• 3.3 – the Scottish Borders Campus is a World Class Centre of Learning
• 4.2 – all young people will be in education, training or employment
• 4.3 – young people demonstrate achievement & attainment
How did we decide?• Examined where SBC PED is responsible
& accountable?– Probably 4 key areas– Looked at the nature of our contribution
• Where SBC contributes with other Stakeholders?– Through Competitive Borders
• Where SBC’s other departments are responsible & accountable– Skills, education, learning
Consultation
• Internal consultation – SBC– Logic Chain Approach
• External consultation– Partners eg SEn, VisitScotland, SDS,
Sectoral Groups– Education sector
• Business Planning
Objective of exercise
• To understand the activities, projects we do
• How these link to strategy, policy, objectives
• And how these need to link to Single and National Outcomes
• What is needed to deliver activities=> And this is what we have come up with
Linking activities to SOA
National OutcomeWe realise our full economic potential with more and better employment opportunities for our people
Local or Single Outcome (2.1)
Sustaining and growing business activity in key sectors of the Borders economy
Likely Activity
- Management of amended Business Gateway- Food & Drink Activity- Tourism activity, including Homecoming 2009- VisitScotland SLA- Procurement Support Project- Events support & planning- Creative Clusters Project
Inputs (£)
Inputs (Human) Business Gateway, Events
Linking activities to SOA
National OutcomeWe realise our full economic potential with more and better employment opportunities for our people
Local or Single Outcome (2.2)
Promoting self-employment and creating sustainable businesses
Likely Activity- Management of Business Gateway contract - SBCC, Export Club support
Inputs (£)
Inputs (Human) Business Gateway
Linking activities to SOA
National OutcomeWe realise our full economic potential with more and better employment opportunities for our people
Local or Single Outcome (2.3)
Maximising employment opportunities
Likely Activity
- Employment Land development - capital bids- Management of building/ premises, tenants- Social Enterprise - Inward Investment, OSB, Film Focus- Employability Coordination- External Funding
Inputs (£)Inputs (Human) Property, Events, Rural Development
Linking activities to SOA
National OutcomeWe live in well-designed, sustainable places where we are able to access the amenities and services we need
Local or Single Outcome (10.3)
Improving the socio-economic performance of our town centres and growing stronger communities
Likely Activity
- Regeneration - capital bids- Regeneration - town centre plans & 3 year project databases- Shop Local - year-around - Employability Coordination- LEADER (Projects & Grants)- SSMEI II- Events - support & planning
Inputs (£)Inputs (Human) Regeneration, Rural & Economic Development
But how do we measure success?
• Outcome = result from outputs – why we should be spending ££ (cumulative = impact)
• Output = The services delivered as a result of activities – measurable
• Activities = The way(s) in which we deliver/ implement
• Inputs = The resources used to deliver or produce activities (& outputs) eg human and financial resources
How to Measure the OutcomeRef Description Indicators Verification
2.1Sustaining and growing business activity in key sectors of the Borders economy
- GVA/ employment in key sectors- SME survival rates in key sectors- Rural productivity
Annual Business Inquiry, BERR, Earnings per head of working age population resident in Borders (Annual Survey of Hours & Earnings)
2.2Promoting self-employment and creating sustainable businesses
- Self Employment rates vs. Scotland- Start Up rates vs. Scotland- Business Stock per WA population
Nomis, BERR
2.3 Maximising employment opportunities
- Employment rate- Claimant Count unemployment rates vs. Scotland- ILO unemployment rates vs. Scotland- ILO inactivity rates vs. Scotland
Nomis, Annual Population Survey
3.1Maximise participation in education, training and employment amongst economically inactive
- Employment rate - Working age population qualifications- Average earnings of employees- Proportion in employment, education or training
- Nomis APS- Nomis APS; Annual Business Inquiry- ASHE- Scottish Government
3.2Existing workforce is highly skilled and responsive to the needs of employers
- Working Age Population qualifications- Earnings- Standard Occupation Classifications- People falling out of work and on to incapacity benefits
- Nomis, APS; ABI- ASHE- Nomis, APS; Scottish Employers Skill Survey- Scottish Government/ DWP
3.3The Scottish Borders Campus Becomes a World Class Centre of Learning
- Learners achieving Level 2, 3, 4+- FE/ HE participation rates- RAE accreditation
- Nomis APS- Scottish Government- HWU/ Scottish Funding Council
4.2All young people will be in education, training or employment
- School leaver destination- FE/ HE participation rates- Vulnerable people in employment, education and training- School exam attainment- Learners achieving entry level qualifications in literacy/ numeracy
- Scottish Government- Scottish Government, Nomis APS- Scottish Government- Scottish Government Destination of School Leavers- Scottish Government - SQA Exam results in Scottish Schools
4.3Young people demonstrate achievement and attainment
- Achievement at English & Maths - Standard Grades- Achievement of Level 1/ 2 qualifications by age 17 or 19- School leaver destination- FE/ HE participation rates by 17 year olds
- Scottish Government SQA results in Scottish Schools- Scottish Government SQA results in Scottish Schools- Scottish Government School Leaver data- Scottish Government
10.3Improving the socio-economic performance of our town centres and growing stronger communities
- TTWA Employment Rates- Population trends- Quality Assessment (vitality & vibrancy)
Nomis, Annual Population Survey, GRO, SBC (Plans & Research)
And what is SBC measuring? • Outcome? Outputs? • Effectiveness in meeting Objectives?• Inputs - Project Managers must consider
the causal link between activities & ultimate performance
• So we use ‘if/ then’ methodology - at each level we can develop targets and indicators
=> For example, the data we need to collect
Example: Business Start Up SupportColumn 1
Objectives Indicators Verification Assumptions
OutcomePromoting self-employment & creating sustainable businesses
ObjectiveTo address barriers to enterpriseTo create sustainable business
Outputs
Marketing & PR of the Business Gateway200 Business Starts pa75% 3-year survival rate
ActivitiesHigh quality management & delivery of the BG service
Critical Condition
Example: Business Start Up SupportColumn 2
Objectives Indicators Verification Assumptions
OutcomePromoting self-employment & creating sustainable businesses
ObjectiveTo address barriers to enterpriseTo create sustainable business
If barriers to business support are lowered, it will result in greater self employment levels & sustaining economic growth
Outputs
Marketing & PR of the Business Gateway200 Business Starts pa75% 3-year survival rate
If these start ups occur, then the overall objective will have been met
ActivitiesHigh quality management & delivery of the BG service
Delivery of BG activity to spec will lead to outputs. There is a risk that the recession will have a greater influence
Critical Condition SE transfer of BG activities
Example: Business Start Up SupportColumn 3
Objectives Indicators Verification Assumptions
OutcomePromoting self-employment & creating sustainable businesses
Self Employment rates vs ScotlandStart Up rates vs ScotlandBusiness Stock per WA population
ObjectiveTo address barriers to enterpriseTo create sustainable business
Business stock per 10,000 WA people vs ScotlandSurvival rates vs Scotland
If barriers to business support are lowered, it will result in more economic activity and economic growth
Outputs
Marketing & PR of the Business Gateway200 Business Starts pa75% 3-year survival rate
# of start up enquiries (tied to promos); # of assists and % converted# of VAT Registrations# of new business bank accounts% of business survival - 12- 36-month
If these start ups occur, then the overall objective will have been met
ActivitiesHigh quality management & delivery of the BG service
Project Inputs: Economic Development Staff and Revenue Budget:Business Gateway - £k;Project Plans;Client satisfaction levels
Delivery of BG activity to spec will lead to outputs. There is a risk that the recession will have a greater influence
Critical Condition SE transfer of BG activities
Example: Business Start Up Support
Regardless of ‘who measures’, there may be issues
• What to monitor – hard & soft outcomes? – Baseline
• Tracking - a data capture strategy; – Primary – Business or Beneficiary – survey?– Secondary – spatial availability? Sampling issues? Delay
• Frequency – over what time period– Longitudinal
• Attribution– Reference Case? Control Groups
• Collection & Ownership– Who and at what cost? – Single point of capture? Shared assessment? Benefits?
Pros & Cons - PrimaryMethodology Benefit Drawback
In project/ programme – data collection
Vital in assessing baseline position against which to measure any change
Requires coordination & resource
Control Informed view of benefits that are attributable to an intervention vs deadweight ie additionality
Complicated, may be costly to recruit and challenge to ensure it is representative
Longitudinal Important to assess longer term effects of intervention; for labour market, allows tracking of changes
Challenge to maintain reliability as non-response rates increase over time; cost
Advice on data used will depend on• Why it needs to be collected; at which level of
the ‘Logic Tree’; and by who• The audience – is it
– Internal (eg Economic Development, budget holders, project managers)
– External (eg partners, funders, public, policy makers)• Their different levels of interest
– Achievements against objectives– Monitoring– Outcomes & Impacts– Value & Lessons– Strategic Added Value