Post on 10-Jun-2015
description
CES 2012 Conference HalifaxTuesday, May 15, 2012 | 15:45‐17:15
Valuing organizational vision in the developmentof performance measurement frameworks
of science‐based organizationsLessons learned from two projects
2
Outline
Context and objective
Background on organizational vision and performance
Overview of two performance framework development projects: Different organization types, visions and projects (tailored)
Effect of organizational vision on the two projects: Approach Methods Results
Project update on framework implementation
Lessons learned For organizations
For external evaluators/consultants
Questions?
3
Context and objective
Research and Technology Organizations (RTOs) increasingly make use of performance measurement (PM) to monitor the achievement of strategic objectives
Approaches to develop performance measurement frameworks are directly aligned with an organization’s:
strategic vision/values
competitive environment
reporting requirements
Using two recent projects, this paper demonstrates how organizational values influenced:
Approach: performance measurement strategies
Methods: how the performance framework was developed
Results: design and implementation of plan for the proposed framework
4
R&D performance measurement
Common goals: Monitor the scientific performance and effectiveness of the
organization on a regular basis Monitor the efficiency and economy of R&D project management and
other supporting processes Inform operational and strategic decisions Feed the production of relevant performance reports Provide reliable information to support effective evaluationCommon challenges: Culture of performance measurement and management Organizational structure and management Internal capacity for planning / performance measurement / evaluation Data availability and quality Buy‐in of data provider and user (championship/ communication)
5
Organizational vision and performance
Many studies have examined the link between the shared vision, values, strategy, goals and practices within organizations (often referred to collectively as ‘organizational culture’) and organizational performance.
Given the difficulty of measuring organizational culture, the research has been inconclusive about the exact nature of this relationship. (Lim, 1995)
However, many researchers have asserted that organizational culture affects all aspects of organizational interactions (Henri, 2006) and that a strong culture is predictive of organizational performance. (Abu‐Jarad, Yusof, & Nikbin, 2010)
Very few studies, however, have looked specifically at how organizational culture influences performance measurement strategies.
6
Organizational vision and performance
Evidence suggests that companies that exhibit more ‘flexible’ or ‘adaptive’ cultures tend to:
have better long‐term performance overall (Abu‐Jarad, Yusof, & Nikbin, 2010; Lim, 1995)
use a greater number of performance measures and focus their performance measurement strategy on supporting decision‐making, justifying activities and establishing priorities. (Henri, 2006)
R&D organizations, in particular, have unique needs and challenges with respect to the capture of impacts
e.g., contributions may be impossible to measure, even in the medium‐long term.
7
Organizational vision and performance
In general, studies suggest that organizations: clarify and translate their vision and values through their PM strategy; align their PM system with their specific R&D profile (basic vs. applied,
short vs. long‐term); use multiple performance measures, including both financial and non‐
financial indicators, to determine short‐term results as well as long‐term processes, impacts and cause‐and‐effect functions;
use their culture as a basis for allocating resources, establishing milestones, and facilitating strategic review and feedback mechanisms.
(Abu‐Jarad, Yusof, & Nikbin, 2010; Jyoti, Banwet, & Deshmukt, 2006; Shahzad, Luqman, Khan, & Shabbir, 2012)
Not surprisingly, we observed the influence of organizational culture as we conducted two projects using two completely different performance measurement approaches that reflected each organization’s vision and values.
8
Overview of two performance framework development projects
Provincial RTO Project Federal RTO Project
Vision: Be a world leader in innovation aligned
with strategic issues of its main client
Approach: Focused +
Participatory
Approach:Exploratory +Comparative
Vision: Be among the top five leading RTOs in the
world
9
Overview of two performance framework development projects
Provincial RTO Federal RTOVision Be a world leader in innovation
aligned with strategic issues of its main client
Be among the top five leading RTOs in the world
Organization type
Provincial RTO that mainly serves the scientific and technological needs of importance for a provincial crown owned company(main client)
Federal RTO that serves thescientific and technological needs of importance for the country
Strategic plan
• 5‐year plan• 4 guiding principles• 5 priorities/11 action items
New plan was in development at that time
10
Overview of two performance framework development projects
Provincial RTO Federal RTOVision Be a world leader in
innovation aligned with strategic issues of its client
Be among the top five leading RTOs in the world
Overall project goal and approach
Goal:• Develop an internal
performance measurement framework to monitor the strategic plan
Expected results: • Actionable plan • Plan developed in
consultation with key staff for buy‐in
Goal:Develop a comparative framework strategy for assessing effectiveness and benchmarking performance against comparable international RTOsExpected results: • A conceptual framework strategy • Creation of linkages with the
international RTO community (for benchmarking)
11
Effect of organizational vision on project approach
Provincial RTO Federal RTO
Vision
Be a world leader in innovation aligned with strategic issues of its main client
Be among the top five leading RTOs in the world
Approa
ch
Focused • Focused on the strategic level• Focused on the identification of
limited/available number of indicators
Participatory• Close collaboration with senior
management (RTO Director General)
• Participatory and interactive approach with internal staff
• PM capacity building and buy‐in
Exploratory• Identify best practices for both operational
and strategic levels• Explore multiple indicators and
measurement approaches Comparative• Consult with other RTOs to compare/
benchmark • Examine how other RTOs are organized and
deliver R&D (client‐focused and impact driven)
• Engage a community of practice
12
Effect of organizational vision on project method
Provincial RTO Federal RTO
Vision Be a world leader in innovation aligned
with strategic issues of its main clientBe among the top five leading RTOs in the world
Metho
ds
Secondary data collection• Focused on provincial RTO’s internal
literature and document/data• Identification of data
availability/feasibilityPrimary data• 4 full‐day roundtable workshops with
different levels of managers: Directors Managers Lead scientists Integration/wrap‐up
• Presentation for validation
Secondary data collection• Extensive scan and review of
document/data on the federal RTO and on the comparable RTOs
• Extensive literature review Primary data• Consultations with some senior
managers from the Federal RTO• Consultation with representatives
(involved in PM and corporate strategy) from 7 comparable RTOs
• Presentation for validation
13
Effect of organizational vision on the project results
Provincial RTO Federal RTO
Vision
Be a world leader in innovation aligned with strategic issues of its main client
Be among the top five leading RTOs in the world
Results
• Report on the proposed/selected indicators and rationale for deployment
• Included alternative indicators for consideration
• Total of 25 indicators (1‐3 per action item)
• 25 page report• All levels of Provincial RTO
management were aware of the project results
• Profiles/analysis of performance measurement approaches and indicators used by other RTOs
• Characterization of operational and strategic indicators
• A strategy to assess the Federal RTO performance
• Step‐by‐step approach for engaging a community of practice (for benchmarking)
• More than 100 indicators explored• 60 page report and 31 pages of technical
material on international RTOs• Federal RTO performance and evaluation staff
were aware of the project results
14
Current status of project implementation …
15
Lessons learned for organizations
Using a focused and participatory approach:
Provides near‐to‐complete product for implementation Limited number of indicators
Data is (almost) readily available
Wider organizational awareness and buy‐in
Provides a clear set of available indicators, but many are proxy indicators at the strategic level
Indicators inform on the progress towards strategic outcomes, but not necessarily on how to improve performance
Potential tension between staff during the definition/selection of operational indicators to measure strategic performance
16
Lessons learned for organizations
Using an exploratory and comparative approach: Necessity to define and conceptualize a full framework, including
Internal measures (operational/corporate‐level indicators) External measures (strategic/impact indicators)
Need to distinguish indicators for internal performance measures and indicators for international benchmarking (can’t always get both)
Need to narrow down the number of potential indicators, especially for international comparison
Access/confidentiality/availability issues of comparable data for benchmark International representatives did not have the same conception of, or
experience with, internal and external measures Implementation requires further challenging steps (final set of indicators,
internal buy‐in, further engagement with RTOs, data sharing agreement, etc.) Organizational culture can impede/influence the selection of common
indicators and how performance is judged
17
Lessons learned for consultants
Using a focused and participatory approach: Data load and complexity more manageable End‐results/tools developed are more transferable to other
projects/RTOs Easy to access data (proximity, language and availability) Greater participation/input from a wider variety of internal
stakeholders Had to help operational managers to think in terms of strategic
indicators Had to mediate differences in performance values from different units No means to compare or calibrate selected indicators with other RTOs
18
Lessons learned for consultants
Using an exploratory and comparative approach: Learned and benefited from the experience of several organizations Further reflection and conceptualization of performance measurement Created links with the international community Challenges to select comparable RTOS and to reach the right
informants The “full comparative matrix syndrome”
Expectation of data in all cells Multiplier effect (50 indicators x 6 dimensions x 8 RTOs = 2,400)
Challenges in the definition and interpretation of collected indicators Language barrier with key PM documents/data sources Much more work compared to the other approach (for similar budget) Had to manage expectations given the ambitious nature of the
international and comparative dimensions
19
Thank you for your time and feedback
CONTACT INFO
Andréa Ventimiglia, BSc MJResearch Analyst | Science‐Metrix
514‐495‐6505 x124andrea.ventimiglia@science‐metrix.com
Frédéric Bertrand, MSc CEVice‐President, evaluation | Science‐Metrix
514‐495‐6505 x117frederic.bertrand@science‐metrix.com
ACKNOWLEDGEMENTSEmmanuel Trépanier MA (contributor)
Eric Archambault PhD (contributor)Julie Caruso MLIS (contributor)
Science‐Metrix 1335, Mont‐Royal E.Montreal, Quebec H2J 1Y6Telephone: 514‐495‐6505Fax: 514‐495‐6523E‐mail: info@science‐metrix.com
WEB SITEwww.science‐metrix.com
Questions?
20
References
Abu‐Jarad, I. Y., Yusof, N., & Nikbin, D. (2010, December). A review paper on organizational culture and organizational performance. International Journal of Business and Social Science, 1(3), 26‐46. Retrieved from http://www.ijbssnet.com/journals/Vol._1_No._3_December_2010/4.pdf
Henri, J. F. (2006). Organizational culture and performance measurement systems. Accounting, Organizations and Society, 31, 77–103. Retrieved from http://arafiki.edublogs.org/files/2011/05/Organizational‐culture‐and‐performance‐measurement‐systems‐1zth1xx.pdf
Jyoti, T., Banwet, D. K., & Deshmukt, S. G. (2006). Balanced scorecard for performance evaluation of R&D organizations: A conceptual model. Journal of Scientific and Industrial Research, 65, 879‐886. Retrieved from http://nopr.niscair.res.in/bitstream/123456789/4952/1/JSIR%2065(11)%20879‐886.pdf
Lim, B. (1995). Examining the organizational culture and organizational performance link. Leadership & Organization Development Journal, 16(5), 16‐21. Retrieved from http://www.emarketing.net.cn/upload/file/2008/06/25/211214400841549.pdf
Shahzad, F. Luqman, R. A., Khan, A. R., & Shabbir, L. (2012, January). Impact of organizational culture on organizational performance: An overview. Interdisciplinary Journal of Contemporary Research in Business, 3(9), 975‐985. Retrieved from http://www.journal‐archieves14.webs.com/975‐985.pdf
21
Theoretical model
J.‐F. Henri / Accounting, Organizations and Society 31 (2006) 77–103
Relationship between culture, use of PM systems and diversity of measurement