Evaluation for Impact and Learning Asia Value Advisors Nov 6 2014
-
Upload
victor-kuo -
Category
Government & Nonprofit
-
view
236 -
download
0
Transcript of Evaluation for Impact and Learning Asia Value Advisors Nov 6 2014
Victor Kuo, Ph.D.郭恩勝
November 6, 2014Hong Kong, SAR, China
EVALUATION IN FOUNDATIONS:ORGANIZATIONAL SYSTEMS FOR IMPACT AND LEARNING
VK
1. Areas of Expertise:• Philanthropy, organizational development and planning• Program evaluation• Education, arts and culture, health, conservation and environment
2. Work Highlights• VK Global Advising, Founder• Seattle Community Colleges, Director of Strategic Planning & Research• Bill & Melinda Gates Foundation, U.S. Education, Program Officers• Western High School, Anaheim, California, School Teacher
3. Education• Ph.D., Stanford University, Educational Administration & Policy• M.A., Columbia University Teachers College, Secondary Science Education• B.A., Pomona College, Biology
ABOUT THE SPEAKER: VICTOR KUO
VK GLOBAL ADVISINGVK
2
Workshop attendees will learn:
1. Approaches to Effective Philanthropy:• Concepts of effective philanthropy, evaluation designs, and evaluation utilization
• Strategies and tools for developing and evaluating projects
• Problems and criticisms about effective philanthropy, evaluating philanthropy
2. Organizing Philanthropic Foundations for Learning and Impact• Definitions and terms of organizational characteristics
• Models for arranging organizations to support learning and impact
• Strategies and tools to facilitate implementation
MAIN POINTS
VK GLOBAL ADVISINGVK
3
I. Approaches to Effective Philanthropy1. Lecture 1: Effective Philanthropy, Concepts and Definitions2. Lecture 2: Evaluation, Concepts and Definitions
a. Activity 1 (individual): Building a Logic Modelb. Summary and questions
3. Lecture 3: Designing evaluations for philanthropic projectsa. Activity 2 (small groups): Developing Evaluation Questions for Different Audiencesb. Summary and questions
4. Summarya. Main points and key wordsb. Questions and criticisms
II. Organizing Philanthropic Foundations for Innovation and Impact1. Lecture 4: Characteristics of Evaluation in Philanthropic Foundations2. Lecture 5: Organizational Tools for Effective Philanthropic Foundations
a. Organization structures, charts and diagramsb. Roles and job descriptions of staff for evaluationc. Budgets and typical costs for evaluationd. Contracts and consultant agreements for evaluatione. Activity 3 (small groups): Developing a “Request for Proposal (RFP)” for an evaluation projects
III. Conclusion
OUTLINE
VK GLOBAL ADVISINGVK
5
I. About the initiative1. Improve graduation rates in US high schools to 80%, especially for low income, minority students
2. From large high schools (1,000 students) to small (300-400 students)
3. School climate 3R’s: rigor, relevance, relationships
4. Improve attendance, progression, and achievement gains
5. More than US$1 billion from 2003-2007
6. 1,900 schools in more than 40 states
7. Chicago, New York City, California, Ohio, Texas, and Washington
8. Networks of school reform models; some research based
II. Foundation strategy 3-levels:1. State: policies2. Districts: alignment and options3. Schools: new school networks and models
III. The problem:1. U.S. high school graduation rates about 60% or less2. Only 32% ready for college or work3. American public schools perceived as broken; unequal
CASE 1: GATES SMALL HIGH SCHOOLS
VK GLOBAL ADVISINGVK
6
I. About the evaluation1. Study timeframe: 2001-2004
2. External research organizations: AIR & SRI
3. Quantitative and qualitative methods; statistical comparison groups
II. Findings1. Schools supported by the foundation had higher levels of rigor, relevance, relationships
2. In math, progress no different than other schools
3. Findings were still early; updates in 2012 show some progress in New York City (70% 4-year
graduation rate)
III. Major challenges1. Teacher turnover
2. Unsupportive policies
3. Class scheduling
4. Inadequate curriculum
5. Lack of a common vision
6. Leadership changes
CASE 1: GATES SMALL HIGH SCHOOLS
VK GLOBAL ADVISINGVK
7
I. About the initiative1. Some of students’ learning is through online delivery of content and instruction2. Student has some control over time, place, pacing3. Learning takes place in a supervised physical location away from home4. Students “rotate” between regular classroom instruction and online instruction at a computer
station5. Foundation invested in 5 charter school organizations; 13 schools
II. Foundation strategy1. Focus on early-stage “Innovation portfolio”
2. Contribute to the evidence base to better understand an emerging learning model
3. Did not want to report a single metric; early comparisons considered inappropriate
III. The problem1. How best to blend online and face-to-face instruction?
2. How to support development of an innovation
3. Opportunity to use technology to improve children’s lives
CASE 2: DELL BLENDED LEARNING
VK GLOBAL ADVISINGVK
8
I. About the evaluation1. Study of schools in first, second, or third year of implementation
2. External research organization: SRI, FSG
3. Quantitative and qualitative methods; statistical comparison groups
II. Findings1. Early signs show students using blended learning outperform those not using it
2. Schools are using blended learning models to personalize learning
3. Teachers reported that students improve procedural skills more than higher order thinking
III. Major challenges1. Integrating online and off-line curriculum
2. Budgeting time and resources for technology infrastructure
3. Need continuous improvement and innovation approach at each school
CASE 2: DELL BLENDED LEARNING
VK GLOBAL ADVISINGVK
9
I. Main points1. U.S. education reform is in an era of achievement2. Philanthropy can support innovation by taking risks and facilitating
learning
II. Challenges1. Education and schools resist change2. How do you introduce and sustain an innovation?3. Implementation
III. Future opportunities1. Disciplined innovation2. Investing in research, development, and evaluation3. Understanding why institutions stay the same and how to change them
SUMMARY
VK GLOBAL ADVISINGVK
11
I. Evaluation in U.S. philanthropy grew rapidly in late 1990’s
1. New philanthropic foundations from business entrepreneurs
2. New thinking on strategic philanthropy
3. Media attention
II. The practice of evaluation in philanthropy has challenges
1. Unclear purpose for evaluation
2. Limited staff capacity
3. Lack of culture of transparency, learning
III. Recommendations
1. Encourage experimentation
2. Build networks
3. Share results; learn from failure
MAIN POINTS
VK GLOBAL ADVISINGVK
12
• News articles about philanthropy grew in the late 1990’s
1. Very large foundations established
2. Wealthy businessmen, celebrities
3. Media attention can be used positively
MEDIA ATTENTION
VK GLOBAL ADVISINGVK
MOTIVATIONS FOR GIVING
2011 UBS INSEAD Study on Family Philanthropy in Asia, p. 24
VK GLOBAL ADVISING 13
• Four types of motivations1. Affiliation
2. Sector interest
3. Impact
4. Pragmatism
• Second generation, entrepreneurs1. Motivated by impact
• Impact Investing1. Financial returns
2. Social returns
3. Environmental returns
VK
14
I. Key Concepts1. Strategy
2. Strategic philanthropy
3. Theory of change
4. Environmental or external scan
II. Historical developments1. Religious, tithing; “spray and pray”
2. 3 M’s: money, modesty, mystery
3. Social impact, value
4. Strategy and management
III. Examples1. Ford Foundation
2. Robert Wood Johnson Foundation
3. Bill & Melinda Gates Foundation
STRATEGIC PHILANTHROPY
VK GLOBAL ADVISING
IV. Tools1. Strategy frameworks
2. Theory of change diagrams
V. Challenges1. Top-down approach too simplistic
2. Adjusting to constant, rapid change
3. Sustaining change requires community
involvement
VI. Future directions1. Strategic philanthropy will continue
2. Balanced with community initiated
approaches
3. Consumer feedback platforms
VK
CONSUMER REVIEW PLATFORMS
Great Nonprofits - 2007
GuideStar - 1994
VK GLOBAL ADVISING 17
Center for Effective Philanthropy
Grantee Perception Reports - 2001
VK
18
I. Key Concepts1. Organizational effectiveness
2. Organizational learning
3. Collective impact
II. Historical developments1. Growth of philanthropic advisories
2. Management and communications experts
3. Systems approaches to social change
III. Examples1. Grantmakers for Effective Organizations
2. David & Lucile Packard Foundation,
Organizational Effectiveness Program
3. Bridgespan, Arabella, RPA, FSG, TCC
4. Center for Effective Philanthropy
EFFECTIVE PHILANTHROPY
VK GLOBAL ADVISING
IV. Tools1. Scorecards, dashboards
2. Learning agenda
3. Systems maps
V. Challenges1. Resistance to “administrative expenses”
2. Difficult to connect organizational
effectiveness to program outcomes
3. Competition among NGOs for funding, few
incentives to work together
VI. Future Directions1. Data visualization for management
2. Crowd-sourcing
3. Collective impact
VK
LEARNING EVENTS
Activities Frequency Participants Goals
Foundation program meetings
4 x each year • Foundation program staff • Align strategies
• Review findings
• Consult experts
Grant portfolio review
1 x each year • Foundation program staff
• Evaluation staff
• Leadership
• Review findings
• Adjust strategy
Grantee conferences
2 x each year • External evaluators
• Grantees
• Review findings
• Adjust programs
VK GLOBAL ADVISINGVK
23
I. Key Concepts1. Evaluation2. Logic model, output, outcome3. Innovation
II. Historical developments1. Social science research, academic2. Community focus, utilization focus3. Management consultants, accountants,
and financial advisors4. Diverse approaches
III. Examples1. Class size reduction studies2. Formative memo, Early College High
Schools3. Annual reports for College-Ready and Post-
secondary education programs4. Infographic
EVALUATION
VK GLOBAL ADVISING
IV. Tools1. Logic model diagrams
2. Evaluation and learning questions
3. List of priority audiences
4. Evaluation Request for Proposal (RFP)
V. Challenges1. Need staff with technical skills
2. Need to balance multiple purposes and
audiences for evaluation
3. Need resources to fund evaluations
VI. Future Directions1. Diverse approaches to evaluation
2. Growth of evaluation internationally
3. Technology, “big data”
VK
24
I. What is it?
• Visual representation, “picture”, of a project
II. Outcomes
• Changes in the condition of target entities (students, schools, district,
community)
III. Outputs
• A unit or a service delivered (students receiving books, teacher workshops
delivered)
IV. Strategies
• Activities conducted to achieve outcomes (a service, research, policy education)
LOGIC MODELS
VK GLOBAL ADVISINGVK
27
LOGIC MODEL TOOLS
Mapping Change: Using a Theory of Change to Guide Planning and Evaluation.
GrantCraft, 2012, in ChineseVK GLOBAL ADVISINGVK
LEARNING QUESTIONS
I. Monitoring
1. What are indicators that show
progress?
2. What early trends are
occurring?
II. Process evaluation
1. How is the project being
implemented?
2. What is supporting or
hindering implementation?
3. What adjustments are needed?
VK GLOBAL ADVISING 28
III. Outcome evaluation1. What is the result of the project?2. How have people’s lives changed?3. What is the added value of the
project?
IV. Innovation1. What is emerging during the
project?2. How are people responding to the
project?3. What about the project should be
changed?
VK
EVALUATION AUDIENCES
I. Leadership
1. Foundation board of directors
2. Foundation executives
3. Program directors
II. Program management
1. Program directors
2. School administrators
3. School teachers
VK GLOBAL ADVISING 29
III. Policy
1. Provincial executive
2. City or town mayor
3. National policy makers
4. National funders
IV. Community
1. Parents with school children
2. Business leaders
3. Local elected officials
4. Civil society organizations
VK
30
• Technical report 1x / year
• Executive summary 1x
• Report brief 1x
• Profiles, vignettes 2 – 4x
• Formative memo 4x
REPORTING OPTIONS
VK GLOBAL ADVISING
• Board presentation 2 – 4x
• Policy brief 1x
• Conference presentations 1x
• Academic journal publication End
VK
STRATEGY AND INNOVATION CYCLES
Bill & Melinda Gates Foundation, The Strategy Lifecycle: A Guide, 2011McKinsey Learning for Social Impact White Paper, Foundation Evaluation Cycle, 2010
FSG From Insight to Action, 2007
31
32
I. Challenges
1. Punishment for negative findings
2. Motivations for giving are not about impact
3. Evaluation is not useful
II. Future directions
1. Cultivate a culture of learning and experimentation
2. Build networks of funders who focus on impact
3. Design evaluations with use in mind
SUMMARY
VK GLOBAL ADVISINGVK
34
I. Characteristics: U.S. Foundations1. 86,192 foundations in US (2013)
2. US$55 billion in giving: foundations
3. US$316 billion in private giving:
foundations, individuals, corporations
4. Types: independent (most), operating,
corporate, community
5. Largest: Bill & Melinda Gates with US$37
billion dollars in assets
II. Issue areas1. Education: US$5 billion
2. Arts and culture: US$3 billion
3. Health: US$6 billion
SECTOR STATISTICS – U.S.
VK GLOBAL ADVISING
III. Largest U.S. Foundations1. Bill & Melinda Gates Foundation
2. Ford Foundation
3. J. Paul Getty Trust
4. Robert Wood Johnson Foundation
5. William & Flora Hewlett Foundation
6. W.K. Kellogg Foundation
7. Lilly Endowment
8. David & Lucile Packard Foundation
9. John D. and Catherine T. MacArthur
Foundation
10. Gordon and Betty Moore Foundation
IV. Education1. U.S. all spending on education: US$1 trillion
2. Gates education spending: US$400 million
(<1%)
VK
35
I. Roles and Titles1. Vice President/Director of Evaluation2. Planning and Evaluation3. Research and Evaluation4. Strategic Assessment5. Impact Planning and Assessment6. Knowledge Management7. Organizational Learning
II. Kinds of evaluation activities:• Evaluation• Performance metrics, indicators• Knowledge management• Research other than evaluation• Aiding in development of program strategy• Grantee satisfaction
III. Evaluation staffing1. About 40 foundations with evaluation staff2. Typically 2-4 staff
CHARACTERISTICS OF EVALUATION
VK GLOBAL ADVISING
IV. Reporting to:• Chief Executive Officer
• Administrator: Chief Financial Officer, Chief
Operations Officer
• Director of Programs
V. History1. Evaluation function in philanthropy greatly
expanded in 1990’s
2. Evaluation grew with shift to “strategic
philanthropy”
3. Phases: research, planning, learning, strategy
VI. Challenges1. Accountability versus learning
2. Managing down versus community driven
3. Internal versus external use
4. “What” to evaluate in a foundation
VK
36
I. Small foundations: 7%• Less than US$50M in grants each year
• 7% of program budget for evaluation
II. Medium foundations: 2%• US$50M - $200M in grants each year
• About 2% of program budget for evaluation
III. Large foundations: 2%• More than $200M in grants each year
• About 2% of program budget for evaluation
COSTS FOR EVALUATION AMONG U.S. FOUNDATIONS
VK GLOBAL ADVISING
I. Project evaluations: 10-15%1. Develop logic model, questions
2. Collect and analyze data
3. Reporting, learning events
II. Donor Engagement: 1-4%1. Design and conduct survey
2. Collect and analyze data
3. Reporting, learning events
III. Scholarship, Donor Advised: 3-7%1. Develop questions
2. Collect and analyze data
3. Reporting, learning events
COSTS FOR EVALUATIONOF SELECT PROJECT TYPES
VK
STRATEGY ASSESSMENT OFFICER JOB DESCRIPTION #1
Responsibilities
1. Contribute to the foundation’s ongoing strategy development and planning
2. Help with the design and implementation of a foundation-wide system to measure social impact, inform learning
and drive improvement
3. Play an integral role in examining new investment opportunities and mapping best practices
4. Develop evaluation tools and metrics in partnership with program teams and grantees
5. Analyze and synthesize research findings to build the foundation’s knowledge management capacity
6. Communicate information gather from funded initiatives to policy makers and non-profits
7. Manage teams of outside vendors and partnersQualifications
8. Masters degree or experience equivalent; advanced degree preferred
9. At least four years of related work experience in strategy consulting, public policy, business, research or nonprofit
management
10. Outstanding analytical skills
11. Excellent communications skills; presentation, written and listening
12. Extremely well organized and strong project management capabilities
13. Bright, energetic, collaborative and goal-driven personality
14. Ability to work independently and take initiative
We are a $2 billion foundation and seek opportunities to build informed, engaged communities through our investments in media, news and information and
through our place-based grantmaking focused on social innovation and community engagement. We are committed to transformational change and the key
elements essential to achieve it – discovery of ideas, belief in a vision, courage to push for change, know-how to get it done and tenacity in getting results.
VK GLOBAL ADVISING37
VK
EVALUATION OFFICER JOB DESCRIPTION #2Responsibilities
1. Manage existing research, tool development, and knowledge management grants
2. Develop, advocate for, and make grants in support of strategic goals (includes soliciting and reviewing proposals,
preparing proposal summaries for director and executive review, and ongoing management of grants made)
3. Performa quantitative data analyses, which support a wide range of efforts within the education team. Contribute
to the team’s knowledge of emerging research and evaluation practices
4. Represent the division in a variety of forums by presenting to policy makers and practitioners in the education
field
5. Business travel typically 25%, but could be higher if needs dictateQualifications
6. A minimum of 5 years experience in the application of qualitative and quantitative research and analysis methods
to social science interventions
7. Masters degree in statistics, mathematics, economics or other quantitative social science; a PhD is preferred
8. Experience developing and executing research plans, selecting and managing outside vendors and solid
knowledge of a broad range of social science research applications, both primary and secondary
9. Experience with secondary schools and an understanding of urban, state and /or national education policy issues
is preferred
10. Demonstrated excellent verbal and written communication skills, with experience presenting to small and large
groups
11. Ability to travel 25% - 50%
VK GLOBAL ADVISING38
VK
39
I. What is a RFP?1. Describes the project background2. Outlines priority audiences, intended use, and preliminary priority questions3. Describes report products (deliverables), timeline and budget parameters4. May include a logic model5. May describe expected methods, data sources6. May describe submission requirements7. May describe applicant scoring and selection process8. May request work samples, client references
II. Typical sections in a proposal1. Context and background2. Evaluator’s professional capacities, qualifications3. Audiences and purposes of proposed evaluation4. Study questions5. Study design6. Data collection and analysis7. Reporting and communications plan8. Establishment of an Evaluation Working Group of client representatives to guide the evaluation9. Budget, timeline, staffing and management plan
REQUEST FOR PROPOSAL
VK GLOBAL ADVISINGVK
40
I. Challenges
1. Developing organizational systems for monitoring, evaluation and learning
2. Organizational structure and culture
3. Demonstrating the value add of evaluation
4. Getting to use
II. Future directions
1. Diverse approaches to evaluation
2. Professional staff with multiple skills
3. Building a supply of philanthropy professionals
4. Establishing connections with related fields: social sciences, evaluation,
accounting, finance, strategy consulting
SUMMARY
VK GLOBAL ADVISINGVK
41
KEY WORDS1. Activities: processes, tools, events, technology and actions that are
part of a program’s implementation
2. Assimilation: process by which a group’s language and culture come to resemble those of another group
3. Centralization: to concentrate authority or power in a central group
4. Charter school: a school that receives public funding but operates independently of local, regional or state school system
5. Collective impact: cross-sector collaboration among organizations with a common vision to achieve social change
6. Comparison group: a group that is exposed to all the same conditions as a treatment group, except for the intervention being tested
7. Control group: a group of untreated targets that is compared with experimental groups on outcome measures in impact evaluations
8. Dashboard: a performance management framework that identifies a few key measures, such as financial, customer satisfaction, internal business processes, and learning and development
9. Empowerment evaluation: a collaborative evaluation in which the evaluator’s role includes consultation and facilitation directed toward capacity building of stakeholders to conduct evaluation on their own and to use evaluation for advocacy, change, and personal change.
10. Evaluation: the systematic investigation of the value of a program or policy
11. Formative: for the purpose of program improvement
12. Generalizability: the extent to which an impact evaluation’s findings can be extrapolate to similar programs
13. Impact: the net effects of a program
14. Indicator: a measure of a concept or social condition
15. Innovation: a new way of doing something; incremental or radical change in thinking, products, process, or organization
16. Input: resources needed to operate a program
17. Institutional theory: a field of sociological thought that examines how and why organizations stay the same, change, and diffuse.
18. Learning organization: an organization that facilitates learning in its members and continuously transforms itself; learning organizations respond to their environment to remain competitive
19. Logic model: a diagram or illustration that shows the connection between planned work and intended results; usually composed of inputs, outputs, and outcomes
20. Social network: a social structure made up of actors, such as individuals or organizations, and the relationships among them
21. Organizational effectiveness: the extent to which an organization is suited to a task or environment; some models emphasize organizations that seek goal attainment, resilience and survival, or adaption.
VK GLOBAL ADVISINGVK
42
KEY WORDS22. Outcome: changes in the conditions of target entities, usually
knowledge, skills, or behaviors
23. Output: direct products of program activities; units of services delivered
24. Philanthropy: “love of humanity”; the use of private wealth to promote social good
25. Qualitative: method of inquiry or investigation that often seeks to answer questions of “why” and “how”, using non-numerical data
26. Quantitative: method of inquiry or investigation that often seeks to answer questions of “what” using numerical data
27. Randomized controlled trial: an impact research design in which experimental and control groups are formed by random assignment
28. Research: process of steps to collect and analyze information to advance knowledge or understanding of a issue
29. School reform: planned efforts to change schools in order to correct perceived social and educational problems
30. Standards-based reform: educational change through specifying skills and levels of competency that students must exhibit, usually through formal tests
31. Strategic philanthropy: approach to philanthropy of not only providing grant support to nonprofits, but to assess social problems, develop strategies to solve them, track the results of their efforts, and use results to adjust future strategies
32. Strategy: a framework for decision-making that is focused on external context and includes a hypothesized causal connection between the use of resources and goal achievement
33. Summative: for the purpose of judgment; to determine if goals were met
34. Theory of change: a theory of how and why an initiative works; describes the pathway of an initiative and the action strategies intended to lead to the achievement of outcomes
35. Treatment group: a group that participates in a program or treatment
36. Utilization focused evaluation: an approach to evaluation where an evaluation is judged based on its usefulness to intended users. In this approach, primary users should be clearly identified, and the evaluator should ensure intended uses guide the evaluation process.
VK GLOBAL ADVISINGVK
43
SELECT REFERENCESArgyris, C. & Schon, D. A. (1974) Theory in practice: Increasing professional professional effectiveness. San Francisco: Jossey-Bass. Ayres, L. (1917).
Cleveland Education Survey, The Cleveland School Survey. Cleveland, OH: The Cleveland Foundation.
Bernatek, B. (2012). Blended Learning: Introduction. Dallas, TX: Michael & Susan Dell Foundation.
Buteau, E., et al., (2009) Essentials of Foundation Strategy. Boston, MA: Center for Effective Philanthropy
Carmen, J. G. & Fredericks, K. A. (Eds.) (2008). Nonprofits and Evaluation. New Directions for Evaluation, 119.
Coffman, J., Beer, T., Patrizi, P., and Thompson, E. H. (2013). Benchmarking evaluation in foundations: Do we know what we are doing? The Foundation Review5(2).
Chelimsky, E. (2001). What evaluation could do to support foundations: A framework with nine component parts. American Journal of Evaluation 22(1), 13-28.
Chen, M. Y. (2006). Developing China's Nonprofit Sector. McKinsey & Co.
Connell, J. P. & Klem, A. M. (2000). You can get there from here: Using a theory of change approach to plan urban education reform. Journal of Educational and Psychological Consultation, 11(1), 93-120.
Fetterman, D. F., Kaftarian, S., & Wandersman, A. (1995). Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks, CA: SAGE Publications.
Field, K. & Holdaway, X. (2013, July 14). A Realm of Influence. Chronicle of Higher Education
Fleishman, J. L. (2007). The Foundation: A Great American Secret. New York: Public Affairs
Fulton, K. & Blau, A. (2005). The Future of Philanthropy: An Orientation for Twenty-First Century Philanthropists. San Francisco, CA: Monitor Group.
Godeke, S. & Bauer, D. (2008). Philanthropy’s New Passing Gear: Mission-Related Investing. New York: Rockefeller Philanthropy Advisors.
GrantCraft. (2012). Mapping Change: Using a Theory of Change to Guide Planning and Evaluation. New York: GrantCraft. (in Chinese)
Kaplan, R. S & Norton, D. P. (1992). The Balanced Scorecard - Measures That Drive Performance. Harvard Business Review (January–February): 71–79.
Kramer, M., Graves, R., Hirschorn, J. & Fiske, L. (2007). From Insight to Action: New Directions in Foundation Evaluation. Boston, MA: FSG Social Impact Advisors.
Kuo, V. (2010). Transforming American High Schools: Possibilities for the Next Phase of Reform. Peabody Journal of Education, 85: 3, 389-401.
Lavizzo-Mourey, R. (2004). Perspective. IN Braverman, M. T., Constantine, N. A. & Slater, J. K. (eds.). Foundations and Evaluation: Contexts and Practices for Effective Philanthropy. San Francisco: Jossey-Bass.
Mahmood, M., Santos, F., Ghaniah, E., & Liao, J. (2011). UBS-INSEAD Study on Family Philanthropy. Singapore: UBS Philanthropy Services and INSEAD.
McKinsey & Company. (2010). Learning for social impact: What foundations can do. London: McKinsey & Company.
VK GLOBAL ADVISINGVK
44
SELECT REFERENCESPatton, M. Q. (2008). Utilization-focused Evaluation. Thousand Oaks, CA: SAGE Publications.
Plantz, M. C., Greenway, M. T., & Hendricks, M. (1997). Outcome measurement: Showing results in the nonprofit sector. New Directions for Evaluation 75, 15-30.
Porter, M. E. & Kramer, M. R. (1999). Philanthropy’s new agenda: Creating value. Harvard Business Review, 121-130.
Powell, W. W. & DiMaggio, P. (1991). The New Institutionalism in Organizational Analysis. Chicago: University of Chicago Press.
REDF Roberts Enterprise Development Fund. (2001). SROI Methodology: A Social Return on Investment, Analyzing the Value of Social Purpose Enterprise Within A Social Return on Investment Framework. San Francisco: The Roberts Foundation.
Rossi, P. H., Freeman, H. W., and Lipsey, M. W. (1999). Evaluation: A Systematic Approach, Sixth Edition. Thousand Oaks, CA: SAGE Publications.
Sanders, J. R. (1997). Cluster evaluation. In E. Chelimsky & W. R. Shadish (Eds.), Evaluation for the 21st Century. Thousand Oaks, CA: Sage.
Scott, W. R. (1992). Organizations: Rational, Natural, and Open systems, third edition. Edgewood Cliffs, New Jersey: Prentice Hall.
Senge, P. M., Kleiner, A., Roberts, C., Ross, R., & Smith, B. (1994). The fifth discipline fieldbook: Strategies and tools for building a learning organization. New York: Crown Business.
Shear, L., Means, B., Mitchell, K., House, A., Gorges, T., Joshi, A., et al. (2008). Contrasting paths to small-school reform: Results of a 5-year evaluation of the Bill & Melinda Gates Foundation’s National High Schools Initiative. Teachers College Record, 111, 1986–2039.
Snow, P. E. & Baxter, L. W. (2002). Framing the big picture: Cluster reviews. Returning Results. Philadelphia, PA: Pew Trusts.
Taylor, F. W. (1911). The Principles of Scientific Management. New York: Harper.
Tyack, D. & Cuban, L. (1995). Tinkering Toward Utopia: A Century of Public School Reform. Cambridge, MA: Harvard University Press.
United Way of America. (1996). Measuring Program Outcomes: A Practical Approach. Alexandria, VA: United Way of America.
Valente, T. W. (1995). Network Models of the Diffusion of Innovations. Creskill, NJ: Hampton Press, Inc.
W. K. Kellogg Foundation. (1998). Logic Model Development Guide. Battle Creek, MI: W. K. Kellogg Foundation.
Wasserman, S., & Faust, K. (1994). Social Network Analysis: Methods and Applications. Cambridge, U.K.: Cambridge University Press.
Wilhelm, I. (2007). Philanthropy’s New Frontier. Chronicle of Philanthropy, September 20, 2007.
Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19(1), 21-32.
Woodwell, W. H. (2005). Evaluation As A Pathway to Learning: Current Topics in Evaluation for Grantmakers. Washington, D.C.: Grantmakers for Effective Organizations.
VK GLOBAL ADVISINGVK
45
Victor Kuo, Ph.D.
郭恩勝
VK Global Advising
Seattle, Washington, USA
CONTACT
VK GLOBAL ADVISINGVK