Post on 29-Mar-2015
Boston | Geneva | San Francisco | Seattle | Washington FSG.ORG
Why Shared Measurement Matters
Srik Gopalakrishnan, FSG
April 2013
2
FSG.ORG
© 2013 FSG
About FSG and Shared Measurement
• Nonprofit consulting and research firm founded in 2000
• Staff of 120 in Boston, San Francisco, Seattle, Washington DC, Geneva
• Success in strategy and evaluation with over 400 foundations, corporations, and nonprofits
• Thought leader: articles published in HBR, SSIR, American Journal of Evaluation including Collective Impact (SSIR 2010)
• Breakthroughs in Shared Measurement and Social Impact (funded by Hewlett Foundation) published in 2009: examined 20 approaches to performance, outcome, impact measurement
FSG is driven by the same passion that drives our clients: a passion for greater social impact using data as a key lever
3
FSG.ORG
© 2013 FSG
The Premise: There Are Several Types of Problems
Source: Adapted from “Getting to Maybe”
Simple Complicated Complex
Right “recipe” essentialGives same results every time
Example: Baking a cake
“Formulas” neededExperience built over time and can be repeated with success
Example: Sending a rocket to the moon
No “right” recipes or protocols Outside factors influence
Experience helps, but doesn’t guarantees success
Example: Raising a child
Traditional approach in social sector has been to treat problems as simple or complicated
4
FSG.ORG
© 2013 FSG
Traditional Ways of Approaching Social Change Are Not Working to Address Our Toughest Challenges
Current approaches not conducive to complex problems:
• Funders select individual grantees that offer the most promising solutions
• Grantees work separately, compete to produce the greatest independent impact
• Evaluation attempts to isolate a particular grantee’s impact
• Large scale change assumed to depend on scaling a single organization
• Corporate, government, philanthropy and nonprofit sectors often disconnected from efforts of foundations and nonprofits
IsolatedImpact
5
FSG.ORG
© 2013 FSG
Imagine a Different Approach
Shift mindset from a “technical” to an “adaptive” approach:
• Understand that social problems – and their solutions – arise from interaction of many organizations within larger system
• Large scale impact depends on increasing cross-sector alignment and learning among many organizations
• Nonprofits, government, philanthropy and corporates actively coordinate their action and share lessons learned
• Progress depends on working toward the same goal and measuring the same things – SHARED MEASUREMENT
IsolatedImpact
CollectiveImpact
6
FSG.ORG
© 2013 FSG
Shared Measurement Is a Critical Piece of Pursuing a Collective Impact Approach
Identifying meaningful common metrics for tracking progress toward a common agenda across organizations, and providing scalable platforms to
share data, discuss learnings, and improve strategy and action
1. Tracking Progress Toward a Shared Goal
2. Enabling Coordination and Collaboration
3. Learning and Course Correction
4. Catalyzing Action in the Field
Definition
Benefits of Using Shared Measurement
7
FSG.ORG
© 2013 FSG
Developmental Phases in Creating a Shared Measurement System
Design Develop Deploy1 2 3
• Shared vision for the system and its relation to broader goals, theory of change or roadmap
• View of current state of knowledge and data
• Governance and organization for structured participation
• Identification of metrics, data collection approach, including confidentiality/ transparency
• Development of data collection tools and technology platform
• Refinement and testing of platform and tools
• Staffing for data management and synthesis
• Learning forums and continuous improvement
• Ongoing infrastructure support
• Improve system based on a pilot, review, refinement, and ongoing evaluation of usability and impact
8
FSG.ORG
© 2013 FSG
Examples of Shared Measurement in Use
9 © 2013 FSG
FSG.ORG
STEM Teacher Experience DRAFT
Excellence Outcomes (quality of teachers)
Candidates demonstrate:• Academic
proficiency• Leadership
potential• Appropriate
mindset to succeed in STEM classrooms and school environments*
Candidates demonstrate basic:• Instructional
practice• STEM content
knowledge• Pedagogical
content knowledge
• Preparedness for the context in which they will teach
Qualified STEM teachers are: • Placed/hired• Prepared for
the context in which they are placed/hired
New STEM teachers demonstrate improved STEM-related:• Instructional
practice• Pedagogical
content knowledge
• STEM Content knowledge
• Student engagement
STEM teachers demonstrate improved STEM-related:• Instructional
practice• Pedagogical
content knowledge
• STEM Content knowledge
• Student engagement
• Student achievement
STEM teachers:• Take on
instructional leadership roles
• Move into administrative roles
• Engage more deeply with community stakeholders
• Influence peers and colleagues
* Mindset persists throughout STEM teaching career
Shared Measures for 100Kin10 Partner Organizations
10 © 2013 FSG
FSG.ORG
Shared Measures for Portland Metro STEM Partnership
11 © 2013 FSG
FSG.ORG
Ways Network Partners Use Shared Measurement System
To Learn with and from Peer Programs and
Organizations
To Inform Internal Decision-making About
Practices
To Identify Practices of High-performing
Organizations
To Identify Other Relevant Partners
To Uncover Areas Where Further Data or Research Is Needed
12
FSG.ORG
© 2013 FSG
Challenges in Developing and Implementing Shared Measurement Systems
Difficulty in coming to agreement on common outcomes and indicators
Concerns about relative performance / comparative measurement across providers working in the same space
Limited capacity (time and skill) for measurement and data analysis within participating organizations
Aligning funders to ask for the common measures as part of their reporting requirements
Time and cost of developing and maintaining a system, both for human capital and technology
Challenges
13
FSG.ORG
© 2013 FSG
Critical Factors in the Development of Shared Measurement Systems Help Overcome These Challenges
Source: Breakthroughs in Shared Measurement and Social Impact, FSG, 2009
Effective Funding Relationships
Strong leadership and substantial funding throughout multi-year development period
Independence from funders in devising indicators and managing the system
Broad engagement in design process by many organizations in field, with clear expectations about confidentiality/ transparency
Voluntary participation open to all relevant organizations
Broad and Open Engagement
Effective use of technology
Ongoing staffing to provide training, facilitation, and to review accuracy of data
Testing and continually improving system through user feedback
Facilitated process for participants to gather periodically to share results, learn from each other, and coordinate efforts
Infrastructure for Deployment
Pathways for Learning and Improvement
14
FSG.ORG
© 2013 FSG
FSG Lessons Learned in Implementing Shared Measurement Systems
Proceed Iteratively
Focus on Learning and Use
Don’t Underestimate the Value of Partner Engagement
Ensure Long-term Funding and Sustainability
Address Both Technical and Political Challenges
A well designed and structured Shared Measurement effort will pay dividends for years to come