Guide to Developing a Risk-Based Departmental Evaluation Plan€¦ · Web viewThe “Revised Risk...

Provincial Government ProgramsCabinet Secretariat, Executive Council

Guide to Developing a Risk-Based

Departmental Evaluation Plan

December 2011

Guide to Developing a Risk-Based Departmental Evaluation Plan

TABLE OF CONTENTS

PURPOSE OF THIS GUIDE............................................................................................2

INTRODUCTION..............................................................................................................3

OVERVIEW......................................................................................................................4Why do Risk-Based Planning?..........................................................................................4Transitional Considerations...............................................................................................5When to Start the Planning Process................................................................................5

GETTING STARTED.......................................................................................................8

WHAT GOES IN A PLAN?..............................................................................................9Policy Requirements............................................................................................................9Five Step Process...............................................................................................................10

GLOSSARY...................................................................................................................18

DOCUMENTS GLOSSARY...........................................................................................21

APPENDIX A: PRE-SCREENING ASSESSMENT TEMPLATE...................................26

APPENDIX B: EVALUABILITY ASSESSMENT TEMPLATE.......................................27

APPENDIX C: EVALUATION PRODUCT PROFILE....................................................28

APPENDIX D: DEPARTMENTAL EVALUATION PLAN..............................................29

APPENDIX E: INITIATIVE INVENTORY UPDATE.......................................................30

APPENDIX F: GOVERNMENT PROGRAM RISK ASSESSMENT TOOL....................31

Cabinet Secretariat │ December 2011 1


PURPOSE OF THIS GUIDE

This document guides departments in developing a risk-based Departmental Evaluation Plan using a five step process and provides templates to assist with the

formatting of the Plan. The five steps are:

The Policy on Evaluation requires that all new and existing government initiatives be

assessed for evaluation priority and that those priorities are documented in a

Departmental Evaluation Plan. New initiatives are assessed for evaluation priority

through the completion of an Accountability Framework1 as part of the Cabinet

submission process. Existing initiatives are assessed for evaluation priority using a

risk-based approach as defined in this document. A web Risk-Based Assessment tool

developed by the Economic and Statistics Branch (ESB) of the Department of Finance

to assist departments in ranking their initiatives from highest to lowest risk. A Guide to

using this tool is contained in Annex F.

Words or terms defined in the Glossary are bolded in their initial appearance in this

document.

1 Developing an Accountability Framework: a Resource and Reference Guide is available through the Provincial Government Programs Office: 729-2590.


1. Compile List of Departmental Initiatives

2. Pre-screen for Exclusion / Inclusion

3. Assess Risk

4. Assess Readiness for Evaluation

5. Submit Plan - Evaluation Product Profiles


INTRODUCTION

The Policy on Evaluation requires Deputy Ministers/equivalents2 to develop a

Departmental Evaluation Plan for approval by the Minister and submission to the Clerk

of the Executive Council. On behalf of the Clerk, the Provincial Government Programs

Office will review departmental evaluation plans and provide comments to the Clerk on

the appropriateness of evaluation coverage including the risk-based approach used to

identify evaluation priorities. The Clerk may refer some or all plans to Treasury Board

or Cabinet for approval.

Plans are to be strategically focused and

founded on an assessment of risk that

the department’s initiatives (programs and services and major policies) may

no longer be relevant, effective or

efficient.

A department has the choice of developing an annual evaluation plan or a multi-year

plan. Multi-year plans are subject to annual review by the department and all plans are

to be monitored and updated as appropriate.

2 For purposes of this Guide, the term department refers to any organization identified in Annex A to the Policy on Evaluation and the term Deputy Minister refers to any head of these organizations.


POLICY PROCESS

Deputy Ministers are accountable for developing and implementing the Plan.

Ministers approve the Plan. Clerk may send the Plan to Treasury

Board or Cabinet for approval.


OVERVIEW

Why do Risk-Based Planning?

The ultimate goal of evaluation is improving client outcomes and ensuring government

initiatives provide value for money (i.e. demonstrating effectiveness, efficiency, and

economy). However not all initiatives need to be evaluated or can be evaluated in a

cost effective manner. The purpose of a risk-based evaluation plan is to ensure

evaluation-related activities are strategically integrated into the department's existing

planning and performance measurement processes and reflect the broader need to

support Cabinet decision-making and budget deliberations. Such planning ensures

resources are allocated to evaluation products (includes evaluations as well as

evaluation-related activities) that can provide the greatest value-added to a department

and Government generally.

As the Deputy Minister is accountable for the development and implementation of the

Plan, it should inform the Minister and Deputy Minister of the scope of the department's

evaluation activities and the limitations of that scope. There should be sufficient

information presented in the Plan to help the Deputy Minister determine if the

evaluation/evaluation-related activities proposed are sufficient and support the priorities

of the Department and of Government.

The best evaluation plan is one that directly supports the information or decision-making

needs of the Minister, Deputy Minister, Cabinet and Treasury Board. It is more than a

list of evaluation-related activities. A good evaluation plan is a tool to:

Engage the Minister and senior executives in identifying the priority evaluation

products required to provide them with information necessary to make informed

decisions;

Communicate the importance of reliable performance data to support monitoring

and evaluation of departmental and government initiatives; and

Identify and secure the resources for completion of the evaluation activities.



Transitional Considerations

The Policy on Evaluation recognizes that departments are not on a level playing field

when it comes to the complexity of their programs and their current evaluation capacity.

The Policy therefore accommodates the capacity discrepancy by allowing for a

transition period to full implementation by March 31, 2014. The composition of a plan is

flexible: plans may be annual or multi-year; there is no expectation that all initiatives be

evaluated, nor is there an established minimum number of initiatives to be evaluated

annually. Deputy Ministers must identify their evaluation priorities and identify what

evaluation products will be completed or commenced during the fiscal year, beginning

with 2012-13. Evaluation products may include evaluation-related training, terms of

reference (TOR) or Requests for Proposals (RFP) for a future program evaluation,

accountability frameworks, research and data analysis, evaluation reports and reviews

as part of a department’s Regulatory Improvement Plan. The identification of evaluation

priorities and the associated resources are the critical elements to be reflected in the

Departmental Evaluation Plan. Some departments may not be in a position to conduct

any evaluations during 2012-13 and may instead engage in capacity building activities

or develop performance measurement plans for priority programs.

As part of building capacity, departments are encouraged to pilot at least one internal

evaluation in 2012-13.

When to Start the Planning Process

To avoid duplication of effort, the planning process can be integrated with the

departmental strategic planning, budgeting and reporting processes. In most cases,

the evaluation planning process should start in the Fall and be completed by early

Spring for the subsequent year's evaluation-related activities (See Table 1). Initial

Departmental Evaluation Plans are to be forwarded to the Clerk no later than April 30,

2012. Departments should identify which evaluation-related activities can be completed

using existing resources and what assistance is needed to complete other product(s).

Ministerial approval is required prior to seeking incremental resources through the



budget process. If such approval is extended, a draft Departmental Evaluation Plan

must be included with the annual budget submission.3

Department’s choosing multi-year evaluation planning shall submit their Plan to the

Clerk at the end of April during the year in which the existing plan expires. All plans

shall be reviewed annually. Where substantive changes to a Plan are required, the

Clerk shall be notified as part of the Deputy Minister’s performance contract process.

Table 1 Central Agency Reporting Schedule

Documentation Jun Jul Aug Sept Oct Nov Dec Jan Feb Mar Apr May

Transparency

and

Accountability

Plans (every three

years)Transparency

and

Accountability

Annual Report* Budget

Submissions

DM Contract

(submission and

reporting)** HR

Accountability

Report

Evaluation Plan

(Department)

* If requesting additional resources to implement the proposed Departmental Evaluation

Plan, a draft of the plan must be attached to the request and submitted with the budget

briefing note. Refer to the Department of Finance’s annual budget guidelines for

process considerations.

** Process and timing under review by the Public Service Secretariat

3 Refer to the Department of Finance’s annual budget guidelines for process considerations.



GETTING STARTED

As stated earlier, the best evaluation plan is one that directly supports the information or

decision-making needs of the

Minister, Deputy Minister,

Cabinet and Treasury Board.

For this reason it is advisable to

meet with the Minister and

Deputy Minister to identify their

information needs, priorities,

and departmental challenges

for the upcoming year.

To complete an evaluation plan

there are multiple documents

that can be helpful in attaining

information. Not all documents

are necessary as the required

information may be accessible

in more than one document. A

short description of documents and how they can be accessed is provided in the

glossary.


Potential Information Sources

Budget Decisions Auditor General Reports

Budget Monitoring ReportsClient Surveys or Public

Consultations

Cabinet or Treasury Board

DirectionsDepartmental Plan (Strategic,

Business, Activity) and Annual

Report

Deputy Minister Performance

Contract

Department Cabinet

Submission Plan

Fiscal Framework to identify

sun setting funds

Performance Monitoring and

Evaluation Reports

Ministerial Mandate LettersPublic Strategy Documents

Operational Plan

Prior Year’s Departmental

Evaluation PlanStrategic Directions

Regulatory Improvement Plan Speech From the Throne and

Budget SpeechesWork Plan


What Goes In a Plan?

Policy RequirementsIn accordance with the Policy on Evaluation, the plan should include:

Specific evaluations or evaluation–related activities (e.g. developing Accountability

Frameworks) as directed by Cabinet or

requested by the Clerk of the Executive

Council following consultation with the

affected Deputy Minister (e.g. evaluations

outlined in the government-wide evaluation

plan4).

Evaluations required by inter-

governmental funding agreements or

legislative requirements;

Specific evaluation activities

required to support the department’s

responsibilities in moving identified horizontal initiatives forward;

Programs for which program funding is set to terminate unless it has already been

determined that an evaluation is not necessary; and

Other direct spending of the department as appropriate to the information needs of

the Minister, Deputy Minister and Senior Management Team.

4 The Clerk will submit to the Premier by March 31, 2012, the initial government-wide evaluation plan.


WHAT GOES IN A PLAN?

Evaluations directed by the Clerk, Cabinet, Federal Provincial Territorial or other Funding Agreements

Evaluations required to support horizontal initiatives

Evaluations required to inform departmental decision making

Programs with sun setting funds


Five Step Process This Guide describes a five step process for developing a Department’s Evaluation

Plan.

1. Compile List of Departmental Initiatives

2. Pre-screening for Exclusion / Inclusion

3. Assess Risk

4. Assess Readiness

5 Submit Plan – Evaluation Product Profiles

Steps 2-4 have associated templates (Appendix A, Risk Assessment Report and

Appendix B) on which assessment decisions are recorded. Each of these completed

templates should be attached to the final Departmental Evaluation Plan.

1. Compile List of Departmental InitiativesDeveloping a plan begins with identifying all departmental initiatives. The PGP Office

requests that departments review with their executive, at the beginning of a

department’s evaluation planning cycle (either annual or multiyear), their inventory of

programs and services to ensure initiatives are accurately reflected, that is

organizational initiatives are listed not organizational structures or functions. Inventories

requiring updating should be submitted to the PGP office using the template in

Appendix E5. Each initiative listed in the inventory should include the associated

budget/cost. Some initiatives do not have a specified budget; in such cases, the cost of

resources to deliver that initiative should be estimated.

Initiatives are defined as a group of departmental activities designed to achieve specific

outcomes. They may be programs or services designed to benefit or influence external

clients, services to internal clients or activities required to monitor policy or program

delivery by third parties (e.g. HCS is responsible for provincial policy with RHAs

delivering the programs). Where a department’s initiative is associated with a

5 Initiatives identified by departments in 2011 have been pre-populated into the risk assessment database by the Economic and Statistics Branch of the Department of Finance. Please contact your assigned Program Officer, Provincial Government Programs Office for information on how to access the web-based tool.



Horizontal Initiative, the initiative should be itemized separately as per Template 6 of

Appendix E.

Corporate Shared Services such as printing services, payroll, purchasing, pensions and

group insurance administration, corporate financial services, various OCIO support

services, etc. should be included in the inventory of the department providing the

service. Departmental support functions/divisions such as Executive Support, Finance

Units, Strategic Human Resource Management Units, Policy and Planning Divisions,

and Communications Divisions should not be included.

2. Pre-Screen for Exclusion / InclusionFrom your inventory there may be initiatives that can be immediately excluded or

immediately prioritized for evaluation (included). The rationale for including or excluding

an initiative at the prescreening stage should be recorded in the standard template

found in Appendix A which will be attached to your Departmental Evaluation Plan

submission.

Exclusion

Evaluations can be time and resource intensive. While departments should have a

performance measurement system in place to regularly monitor and report on program

performance, evaluations should only be undertaken on a periodic basis and only if the

benefits of the evaluation are expected to exceed the costs of completing an evaluation.

Accordingly, initiatives that have been evaluated in the last two years can be excluded

from the next year’s departmental evaluation planning.

Other initiatives may become excluded after completing the Evaluablility Assessment in Section 5.2.4 of this document.



Inclusion For some initiatives, there may already be a mandatory requirement for an evaluation.

The source of the requirement may have been through Cabinet or Treasury Board

direction, initiatives identified under Accountability Frameworks (for new initiatives or as

part of an Horizontal Initiative), initiatives requiring funding renewal (as specified in

budget decisions or MCs), legislative requirements6, requests by Ministers or Deputy

Ministers, or where other funders are involved (e.g. the federal government) as a

requirement of the Contribution Agreement. Those initiatives with mandatory evaluation

requirements do not require a Risk-Based Assessment but do require an Evaluability

Assessment under Section 5.2.4 of this document.

Reviews or evaluations identified in a department’s Regulatory Improvement Plan

should be documented in the Departmental Evaluation Plan but do not require

assessment for risk or evaluability.

3. Assess RiskAll department initiatives remaining after the pre-screening process must be assessed

for risk. The probability that a program is no longer relevant, effective or efficient is the

overall risk that is being measured. The Economic and Statistics Branch (ESB) of the

Department of Finance has developed a web Risk-Based Assessment tool to assist

departments in ranking their initiatives from highest to lowest risk.7 The tool’s ranking is

based on the following seven risk variables with predefined weighted sub criteria. The

attached Risk Assessment Tool Guide will provide step by step instructions for

completing the on line tool.

Budget: This risk factor considers materiality of the base budget. Initiatives utilizing a

significant percentage of a department’s resources may be evaluated periodically to

6 Legislative requirements refer to any legislation requiring periodic reviews of initiatives. It does not refer to reviewing the legislation itself. 7 Please contact your Program Officer to determine how to access this web-based resource.



confirm value for money (effectiveness, efficiency and economy).8 Helpful documents

may include: Budget Decisions, Estimates.

Gross Expenditure Variance: This risk factor considers the significance of over or

under spending trends. Initiatives experiencing significant over or under spending

should be evaluated periodically to confirm adequacy and relevance (i.e. Are existing

resources adequate to address the identified population’s need; is the initiative over-

resourced based on existing needs). Helpful documents may include: Budget

Monitoring Reports, Report on the Program Expenditures and Revenues of the

Consolidated Revenue Fund.

Priorities: This risk factor considers how important the initiative is to achieving a

government-wide priority. Helpful documents may include: Prior year’s Departmental

Evaluation Plan, Ministerial Mandate Letters, Speech from the Throne, Public Strategy

Documents, Deputy Minister Performance Contract, Departmental Plan (Strategic,

Business, or Activity), Annual Report, Operational Plan, and Work Plan.

Funding Renewal Commitments: This risk factor applies where evaluation may

support a request to continue funding that was approved for a fixed period or for which

there is no continuing provision in the fiscal framework. Helpful documents may

include: Fiscal Framework, Budget Decisions.

Public Opinion: This risk factor considers client and other stakeholder feedback about

an initiative. There may be current sensitivities, interests or pressures from clients and

other stakeholders that would warrant an evaluation (e.g. feedback from the Auditor

General, individuals, groups or MHAs). Helpful documents may include client surveys,

public consultations, Auditor General Reports, public strategy documents, departmental

correspondence such as letters to Ministers, etc.

8 ESB led a group of Policy Directors and PGP staff through a multiple criterion evaluation process to assign weights to the 7 factors. As a result of this process, the Budget factor is not statistically significant. Departments may choose to include initiatives which use significant resources in their Plan even if assigned a low risk assessment using this tool.



Relevance: This risk factor considers the results of a department’s environmental scan

and SWOT (strengths, weaknesses, opportunities, threats) analysis. Departmental

strategic planning and/or horizontal strategy development may have identified changes

in the demographics and/or needs of the target population that requires a revised needs

assessment, process review and/or effectiveness evaluation (e.g. growing wait lists for

services, lower than expected take-up). Helpful documents may include: Auditor

General Reports, Performance Monitoring and Evaluation Reports, Departmental Plan

(Strategic, Business, or Activity) and Annual Report.

Performance Findings: This risk factor considers the findings from the performance

monitoring and or past evaluation reports. Helpful documents may include:

Performance Monitoring and Evaluation Reports, Auditor General Reports.

Upon completing the web Risk-Based Assessment Tool for all initiatives, a final report

can be automatically generated ranking initiatives from 1 (highest risk) to 0 (lowest risk).

The final Risk Assessment Report is to be reviewed and approved by executive and

attached to the Departmental Evaluation Plan submission.

There is no identified “high risk” cut-off point. Initiatives will cluster differently along the

continuum across departments. For example some departments’ initiates may cluster

primarily in the orange to red zone while others may cluster closer towards the blue

spectrum. In either case departmental executive must choose a Risk Level Cut-Off

point (between 1-0). All departments will be expected to identify Evaluation Products

within the planning cycle. Those initiatives close to or above the cut off point must be

assessed for evaluability (i.e. which priority initiatives are able to have an evaluation

product addressed in this planning cycle).



4. Assess Readiness The risk assessment process identifies which initiatives are potential priorities for

evaluation: the evaluability process identifies which priorities are feasible to evaluate in

the current planning cycle. Initiatives to be assessed for evaluability include those

under Section 5.2.2 assessed as mandatory and those initiatives that have been

identified as priority due to their proximity to the Department’s Risk Cut-Off point

(Section 5.2.3). All Evaluability Assessment outcomes should be recorded in the

standard template found in Appendix B. Each Evaluation or Evaluation Activity to be

completed in this planning cycle must have a corresponding Evaluation Product Profile

(Appendix C) completed to describe what is expected to be completed by when and at

what cost. The evaluability process is defined by three key questions as illustrated in

Chart 1.

Chart 1 Process Flow Chart: Priority Evaluation Initiatives



Question 1: Will the initiative likely be modified based on evaluation results? Department leads should consult with their Minister and Deputy Minister to determine if

there is an ability to modify the initiative based on findings. If there are timing issues

which make evaluation in the current planning cycle not feasible, the initiative should be

kept in view and reassessed in future planning cycles.

Questions 2: Is the initiative ready to be evaluated? Preconditions for evaluating the effectiveness and efficiency of initiatives include:

a clearly articulated initiative

clearly specified goals and outcomes

a rationale linking the initiative’s activities to the desired outcomes and

appropriate performance measurement data

If an initiative cannot be evaluated, the departmental evaluation plan should identify

what Evaluation Activity is required to position the initiative for evaluation at a future

date (e.g. developing a logic model, implementing a performance measurement plan,

gathering baseline data, and so forth). These Evaluation Activities must be detailed in

an Evaluation Product Profile, Appendix C.

Question 3: Are resources available to complete the evaluation?Evaluation activity planning will require assessing internal/external resource

requirements. Where existing departmental resources are inadequate to cover all

priority evaluations, those that can be deferred should be kept in view for consideration

in subsequent evaluation cycles. Those that cannot be deferred and would require

incremental funding should be brought to the Deputy Minister’s attention. The Minister

and Deputy Minister must determine if existing resources can be reallocated or consider

the budget guidelines to determine if they will seek additional resources through the

budget process9. If incremental resources are being requested, a draft of the

Departmental Evaluation Plan should be attached to the budget submission.

9 Departments need to consider the Department of Finance budget guidelines when determining if additional resources should be sought through the budget process.



The Economic and Statistics Branch (ESB) is a resource available to all departments.

The ESB may be able to assist in gathering baseline data, developing data from

administrative or other sources, ensuring datasets are suitable for an evaluation and/or

developing or assessing potential measurement tools for an evaluation. They can

assist in making sure the data and tools needed for an evaluation are available when

needed by considering requirements early in the process. If there will be incremental

cost to the ESB to provide the support, there may be a charge to the department which

should be identified in the budget submission along with any other incremental costs

associated with the evaluation.

5. Submit Plan – Evaluation Product Profiles Once the Minister and Deputy Minister have been briefed and their input obtained, a

final Departmental Evaluation Plan should be prepared. A standard template is

presented in Appendix D. For each proposed evaluation product, an overview should

be included reflecting: critical timelines (at a minimum expected start and end dates);

resource requirements including whether these would be internal or external; and `the

department lead overseeing the activity. A standard template to be completed for each

evaluation product profile is presented in Appendix C.

The final plan should be forwarded to the Minister for approval. As per the Policy on

Evaluation, the approved plans must be forwarded to the Clerk of Executive Council by

April 30, 2012 for the initial plan, and in subsequent years as plans are scheduled for

renewal. The Clerk may submit the Departmental Evaluation Plan to Treasury Board or

Cabinet for further consideration.

The Clerk will be advised of the department’s progress towards implementation of the

plan as part of the Deputy Ministers’ performance reporting process.



GLOSSARY

Accountability Framework: The Policy on Evaluation requires Deputy

Ministers/equivalents to ensure all Cabinet submissions for new policies and programs

identify the client, other benefits, and where feasible, establish specific, measurable,

achievable, realistic, and time bound (SMART) targets before decisions on

introducing/funding are made by Cabinet. It also requires them to “ensure Cabinet

submissions include an accountability framework documenting how performance will be

monitored and determining if and when a more comprehensive evaluation of the

program/policy will occur”. An Accountability Framework makes roles, responsibilities,

and expectations clear, supporting the availability of reliable and timely reports about

intended and actual results. The Accountability Framework structure includes

performance-based planning, monitoring, performance-based reporting and feedback.

Accountability Frameworks are attached to applicable Cabinet papers submitted since

July 1, 2011.

Consolidated Revenue Fund: is the account into which public money (taxes and

revenues) is deposited, and from which funds are withdrawn in order to pay the costs of

public services. The Legislature authorizes the use of the money in the Consolidated

Revenue Fund.

Departmental Evaluation Plan: is a clear and concise summary that establishes the

department's evaluation activities / products to be undertaken over a specified period, in

accordance with the Policy on Evaluation.

Economy: minimizing the use of resources. Economy is achieved when the cost of

resources used approximates the minimum amount of resources needed to achieve

expected outcomes (i.e. attaining the right program inputs at the lowest possible costs).

Effective: the extent to which the observed outcomes are consistent with the intended

outcomes.



Efficient: the extent to which resources are used such that a greater level of output is

produced with the same level of input or a lower level of input is used to produce the

same level of output (i.e. attaining the most program outputs possible for each program

input). The level of input and output could be increases or decreases in quantity, quality

or both.

Evaluability Assessment: is used to determine if an initiative is able to be evaluated in

the current planning cycle. An initiative is evaluable if: it is likely be modified based on

evaluation results; the initiative can be clearly and logically articulated, it has

performance data or can be developed using evaluation methods and there are

resources available to complete the evaluation.

Evaluation: the systematic collection and analysis of information on the performance of

a policy, program or initiative to make judgments about relevance, progress or success

and cost-effectiveness and / or to inform future programming decisions about design

and implementation. Types of program evaluations include: Evaluability Assessment,

Needs Assessment, Monitoring / Review compliance with performance standards,

Implementation / Process Evaluation (Formative Evaluation), Impact/Outcome

Evaluation (Summative Evaluation), Program Review and Efficiency Assessment

(Cost/Benefit Analysis or Cost Effectiveness Analysis).

Evaluation Related Activity: any output of the departmental evaluation function which

may include, but is not limited to, the following: training, terms of reference (TOR) or

Requests for Proposals (RFP) for a future program evaluation, accountability

frameworks, research and data analysis, and reviews monitored under a department’s

Regulatory Improvement Plan.

Evaluation Product: Evaluations and evaluation-related activities.

Horizontal Initiative: An initiative is considered horizontal if multiple provincial

ministers are accountable for the results of the initiative at this point in time. An

initiative’s accountability structure may change over the course of its life cycle.



Committees of multi-departmental initiatives should conduct periodic reviews over the

initiative’s life cycle to determine the appropriate level of accountability.

Performance Measure: a quantitative or qualitative standard that characterizes and

defines initiative’s results including outputs and outcomes (e.g. the “success standard”

by which you know there has been a change in knowledge, skill, behavior, or attitude).

Performance Measurement: the process of selection, development and ongoing use

of performance measures for program management or decision-making. Generally

performance measurement occurs in the form of quarterly or annual monitoring of

performance measures to determine if the desired change is occurring. Performance

measurement can identify the direction of change but does not attribute the cause of the

change to the initiative.

Performance: the actual results measured against defined standards (i.e. the extent to

which effectiveness, efficiency and economy are achieved by a program).

Policy: A policy enables or restricts actions as a means of specifying outputs,

outcomes, or parameters. Policy consists of official guidelines or operating principles

that influence behaviour towards a stated outcome.

Program: a group of related purposeful activities that is intended to achieve one or

several related objectives. A program is often treated as a budget unit, that is designed

and managed to meet a specific public need either directly or indirectly (that is, supports

the operations of government).

Relevant: the extent to which a program responds to a demonstrable need of

Newfoundlanders and Labradorians and is appropriate for delivery by the provincial

government.

Risk-Based Approach: is a method for considering risk when planning the extent of

evaluation coverage of direct program spending / policy initiatives. The risk being

measured is that an initiative may not be, or is no longer relevant, effective or efficient,



Value for Money: the extent to which a program demonstrates relevance and

performance (effectiveness, efficiency and economy). It is an important goal for public

officials and other stakeholders who are concerned with whether taxpayers and citizens

are receiving effective and efficient programs and services for their tax dollars.

DOCUMENTS GLOSSARY

Auditor General’s Reports: As a legislative auditor, the Auditor General audits

financial statements and other accountability documents, evaluates management

practices and control systems, and determines compliance with legislative and other

authorities. The Auditor General reports at least annually to the House of Assembly on

significant matters which result from his/her examinations.

Budget Speeches: An itemized summary of estimated or intended expenditures for a

given period along with proposals for financing them. Annual budget speeches are

accessible on the House of the Assembly publications website. The Budget combines

the projected accrual revenues and expenses of the Consolidated Revenue Fund with

the budge of various Crown Corporations, Boards and Authorities which are controlled

by the Government of NL.

Budget Decisions: Subsequent to Cabinet decisions on department’s budget requests,

the Budget Division of the Department of Finance will issue documents recording the

decisions. From time to time, there may be a requirement for a department to submit

additional information or report back to Treasury Board or Cabinet on specific issues.

These documents are filed with Deputy Ministers and with Department Comptrollers.

Budget Monitoring Reports: On a monthly basis, Departmental Controllers submit to

the Budget Division of Finance, a report highlighting any variances between projected

expenditures to March 31st of each year as compared with the original budget.

Departmental Controllers will have copies on file.

Cabinet or Treasury Board Directions: This category includes minutes and other

correspondence issued by Cabinet or Treasury Board to a department directing a



specific action. These documents are generally in the form of a Minute of Council (MC);

Order in Council (OC) or a Treasury Board Minute (TBM). These documents are filed in

the minister’s office.

Client Surveys or Public Consultations: Many initiatives consult with citizens in the

development stage or report progress at incremental stages. Feedback from

consultations is generally summarized and used to improve on program performance.

This feedback will be valuable in answering the risk-based assessment public opinion

question and may be relevant in answering question 1 of the evaluability assessment.

Oftentimes public feedback is included in public reports (e.g. “What We Heard”.

Program Managers will be able to identify where this information can be found).

Departmental Plan (Strategic, Business, Activity) and Annual Reports: The

Transparency and Accountability Act requires all government entities to submit to

Cabinet a planning document. Depending on the entity’s categorization (category 1, 2,

or 3) these planning documents are referred to as Strategic, Business or Activity Plans

respectively. Departments are required to report annually to Cabinet on their plan’s

progress (i.e. Annual Reports. All plans and annual reports are public documents and

are posted on the department’s website).

Department Cabinet Submission Plan: To facilitate Cabinet scheduling, departments

are required to identify what issues they intend to bring before Cabinet, with a brief

description and an anticipated submission date. As the Policy on Evaluation requires all

Cabinet submissions requesting major new policies or new initiatives to have an

accountability framework, the Cabinet submission is a priority document when

developing an evaluation plan. The plans are highly confidential and are filed in the

department’s ministerial office. Cabinet Submission Plan information may be accessed

through department Policy Directors.

Deputy Minister’s Performance Contract: The Deputy Minister reports to the Minister

and the Clerk of Executive Council in ensuring that all over-arching policies of

government are communicated and coordinated, and that matters and sensitive issues



requiring strategic direction on the part of government are monitored to ensure their

expeditious resolution. Each Deputy Minister has a performance contract which is

reviewed periodically with the Clerk in consultation with the Minister. These Contracts

are highly confidential and are filed in the Deputy Minister’s Office and with the Clerk of

Executive Council. Departmental policy divisions draft these contracts and can ensure

all pertinent information needs are met by the proposed plan.

Estimates of the Program Expenditure and Revenue Fund: is a supplementary

document to the Budget. It outlines the estimated expenditures and revenues of the

Consolidated Revenue Fund for the year and reflects the policies, programs and

priorities of government. This document is available on the Internet:

www.fin.gov.nl.ca/fin/public _accounts/index.html .

Fiscal Framework: While Estimates are prepared for a single fiscal year, the Budget

Division of the Department of Finance maintains a multi-year forecast of departmental

expenditures and revenues by budget categories (e.g. salary and operating envelopes,

major programs). A quick review of this Fiscal Framework can identify budget

adjustments such as the removal of funds for one-time initiatives or those being piloted.

Ministerial Mandate Letters: A letter oftentimes issued by the Premier to his/her

Ministers outlining the government priorities that fall within a ministry’s mandate. Such

letters would be filed in the Minister’s Office.

Operational Plan: The operational plan focuses on the key priorities of the Deputy

Minister which relate to the internal administration of the organization (i.e. monitoring

and evaluating organizational performance, organizational structure, development and

implementation of organizational policies and procedures). Operational plans are

encouraged by the Transparency and Accountability Office and, as such, department

Accountability Coordinators will be of assistance in accessing operational plans.

Performance Monitoring and Evaluation Reports: Many programs have undergone

monitoring and/or evaluation. Understanding the monitoring and evaluation history,


http://www.fin.gov.nl.ca/fin/public%1F%1F_accounts/index.html


including recommendations for program changes, will facilitate risk-based assessments.

For example, programs that have been evaluated in the last two years are excluded for

consideration under the current year’s risk-based assessment. However, these

monitoring or evaluation reports may make recommendations regarding future

monitoring and evaluation needs. These reports come in many forms and may be

generated internally or externally. Program Managers and Directors may be of

assistance in tracking such reports.

Public Strategy Documents: Strategic documents outline a plan of action designed to

achieve a particular goal. Such documents are used by departments to clarify an

approach to an important broad-based issue (e.g. wellness, climate change, innovation,

and so forth). These reports are helpful in identifying the various initiatives that are

covered under the strategy. Most departments post their strategy documents on their

website. Policy Directors can facilitate access to such documents if they are not readily

available.

Prior Year Departmental Evaluation Plans: Evaluation plans illustrate the decision

making process used to define priority initiatives for evaluation. An understanding of

these processes will facilitate identifying current priority initiatives. Evaluation Plans are

not public documents. They can be accessed through Policy Directors.

Regulatory Improvement Plan: All Departments are required under the Regulatory

Reform Initiative to submit an annual or multi-year plan to Service Newfoundland and

Labrador. Quarterly progress reports by departments demonstrate qualitative and

quantitative improvements in regard to developing and implementing regulation.

Review-related activities monitored under this plan are considered to be an “evaluation

product” and should be credited as such in the Departmental Evaluation Plan.

Regulatory Improvement Plan initiatives do not require Risk or Evaluability

Assessments under the Policy on Evaluation. All departments have a Regulatory

Reform Representative who can advise on, as applicable, the initiatives included within

the department’s Regulatory Improvement Plan.



Speech from the Throne: The Speech from the Throne occurs at the opening of a

session of the House of Assembly. The Speech is read by the Lieutenant Governor-in-

Council and prepared by the Premier's Office. Speeches are publicly accessible on the

House of the Assembly publications website.

Strategic Directions: Strategic direction means the articulation of a desired physical,

social or economic outcome that would normally require action or involvement of more

than one government entity. They are a summary of government’s commitments to the

people of the province. They are communicated by Government through Platform

Documents, the Speech from the Throne and Budget Speeches, press releases, and

policy documents. Each minister has his/her own strategic directions. Accountability

Coordinators play an essential role in the development of the Strategic Directions for

their respective ministers. The strategic directions of a department are contained in the

Department’s Plan (strategic, business, activity).

Work Plan: Work plan issues may emanate from the department plan, the operational

plan, or reflect priorities at a branch, division or position level. The work plan contains

the work direction of a specific branch, division or position. Divisional Directors will be

of assistance in accessing work plans.



APPENDIX A: Pre-Screening Assessment Template

From your inventory there may be initiatives that can be immediately excluded or

immediately prioritized for evaluation (included). The rationale for including or excluding

an initiative at the prescreening stage should be recorded in template 1. This template

is to be attached to your Departmental Evaluation Plan submission.

Template 1 Pre-Screen Inventory

PRE-SCREENING OUTCOME

InitiativeMandatory Inclusion

CommitmentExclusion

Evaluated in the last two yearsInitiative W MC2009-xxxxInitiative X Ministerial PriorityInitiative Y 2010Initiative Z Poverty Reduction Strategy

Evaluation



APPENDIX B: Evaluability Assessment Template

The Template 2 below records evaluability decisions for all priority initiatives. Priority

initiatives include those identified through the Pre-Screening: Mandatory Inclusion

(Section 5.2.2) and those initiatives that have been identified as priority relative to the

Department’s Risk Cut-Off point.

Using Template 2, list the initiatives that have been assessed for “Evaluability” and

check the box identifying the assessment outcome. Where “Other” has been checked,

please add a footnote to provide an explanation for the choice.

Template 2 Evaluability Assessment Template

Initiative

EVALUABILITY ASSESSMENT OUTCOMES

*Evaluation To Be Completed In This Planning Cycle

*Evaluation Activity To Be Completed In This Planning Cycle:

No Evaluation Or Evaluation Activity To Be Completed In This Planning Cycle:

Not Ready For Evaluation

Insufficient Capacity (e.g. need staff training, data bases developed etc)

Other Initiative Unlikely To Be Modified Based On Evaluation Results

Insufficient Resources

Other

W X Y Z

* An Evaluation Product Profile (Appendix C) is to be completed for each initiative that indicates a check under the categories of Evaluation or Evaluation Activity.


APPENDIX C: Evaluation Product Profile

The Evaluablity Assessment Outcomes (template 2) identify all Evaluations and

Evaluation Activities that are to be completed within this evaluation planning cycle. For

each planned Evaluation and Evaluation Activity, an Evaluation Product Profile

(template 3) is to be completed to describe what is to be completed, by when and at

what cost.

Template 3 Evaluation Product Profile

INITIATIVE TITLE:

Brief description of the initiative and the scope of the evaluation product to be undertaken.

Proposed Timeframebegin date: end date:

Who is responsible to oversee this work?

Resource Requirements $ Within Base *Budget Request$ Reallocated $ New

InternalExternalEconomic Statistics Branch

Total

* Please refer to the Department of Finance’s annual Budget Guidelines to determine the parameters under which new funding can be requested.

APPENDIX D: Departmental Evaluation Plan

Template 4 Departmental Evaluation Plan

1. INTRODUCTION1.1 Department’s Strategic Directions and Goals:

Provide a brief discussion of how evaluation is being used to support the department’s mandate.

2. ASSESSMENT DOCUMENTS 2.1 Pre-Screening Outcomes

Attach the completed Appendix A Template 12.2 Risk Matrix Outcomes

Attach the web-based Risk Assessment Report 2.3 Evaluability Assessment Outcomes

Attach the completed Appendix B Template 23. YEAR IN REVIEW3.1 Evaluation Products Conducted in Year XXXX-XXXX :

This section should identify evaluations and evaluation products that your department has completed in the previous evaluation planning cycle. The Provincial Government Programs Office has developed an evaluation inventory for each department for the period 2009-2011. Please contact your PGP Officer for a copy.

3.2 Evaluation Activities to Be Carried Forward: This section should identify evaluations and evaluation products that your department had anticipated completing in the previous planning cycle but are required to carry over into the next planning cycle.

4. CURRENT YEAR’S WORK PLAN4.1 Proposed Evaluation Product Profiles:

Complete Template 3 in Appendix C for each initiative for which there is an evaluation product to be addressed in the current year’s evaluation cycle.

4.2 Proposed Timeline for Evaluation Activities for Current Year: Summarize the evaluation products to be delivered in this evaluation cycle through a time line.

4.3 Resource Requirements:Summarize the resource requirements identified in the individual product profiles

4.4 Regulatory Improvement Plans (RIP):Evaluations/Product commitments associated with RIP should be noted in this section and cross referenced to the RIP.

APPENDIX E: Initiative Inventory Update

Departmental initiative inventories and their budgets should be reviewed, with

executive, at the beginning of each evaluation planning cycle to ensure initiatives are

updated and accurately reflected (i.e. new initiatives, initiatives removed, and ensure

organizational initiatives are listed not organizational structures or functions).

Templates 5 and 6 should be completed and submitted to the PGP Officer for

population into the web-database. The templates below provide examples.

Template 5 Departmental Initiative Inventory Update

Department Initiative: Component * Estimated 2011 Budget

% of Gross Dept Budget

Health and Community Services

NL Provincial Drug Program: Access Program

FinanceTax Expenditures: PIT - Child Benefit

Where a department’s initiative is associated with a Horizontal Initiative, this association

should be identified in the inventory as illustrated in template 6.

Template 6 Horizontal Initiative Inventory Update

Horizontal Initiative (HI)

HI Lead Department

Sub-Initiative (SI) SI Department

* Estimated Budget for Year 20XX

Provincial Waste Management

Environment and Conservation

Policy EC $ 40,000Implementation MA $50MEnforcement SNL $100,000

* If the initiative does not have its own Cost Centre in ORACLE GL, please provide an order of magnitude estimate of the budget resources it consumes (can be rounded to nearest $100,000 or 1/2 million)

Estimated 20xx Budget / Total Gross Current Account Expenditures per Estimates 20xx

APPENDIX F: Government Program Risk Assessment Tool

USING THE RISK-BASED ASSESSMENT TOOL

The Guide Developing a Risk-Based Departmental Evaluation Plan, Step 3, discusses

the process for completing a risk-based assessment. All department initiatives

remaining after the pre-screening process must be assessed for risk. The probability

that a program is no longer relevant, effective or efficient is the overall risk that is being

measured. Step 3 of the Guide references an online web Risk-Based Assessment tool

developed by the Economic and Statistics Branch (ESB) of the Department of Finance

to assist departments in ranking their initiatives from highest to lowest risk. The tool’s

ranking is based on seven risk variables with predefined weighted sub criteria. The

information contained in this appendix will aid employees in navigating through the web

Tool.

The tool can be accessed using the following address. Please open your internet

browser and copy and paste the following - http://inform.stats.gov.nl.ca/riskassessment.

A single user ID and password has been assigned to each Department but more than

one person can access the database at one time. Once logged in you will see the

following screen.

http://inform.stats.gov.nl.ca/riskassessment

Screen 1

Screen 1 provides three tab options. The “Add Risk Assessment” button is used each

time you go to complete the initial risk assessment. The “Revised Risk Assessment”

button is used to make changes to an already completed risk assessment. The “View

My Projects” button allows you to view and export to Excel or Word all risk assessments

completed by Project.

To begin a new risk assessment, click on Tab “Add Risk Assessment” and you will get

the following screen:

Screen 2

A “Select Project” and “Program Names” appears on Screen 2. The “Select Project” tab

has been pre-populated using the Departmental abbreviation and 2012 and the

associated program names have been similarly pre-populated based on information

provided to the Provincial Government Programs Office (PGP). The PGP can add new

Projects (for example if a lead for a Horizontal Initiative decides to use this tool to assist

in developing an evaluation framework for their strategy) and or new program names.

In this illustration it presents the list of programs for CS2012 (Cabinet Secretariat)

Program Description as well as a tab for each of the seven risk factors. When printing

out reports it is the “Select Project” identifier that will be used. All risk assessments

completed under your department’s Project name will be available for exporting and

printing.

To complete a risk assessment, scroll through the list of program names and select the

one you want to assess. If it does not exist, please contact the PGP. Once you select a

program and click NEXT, you will advance to the following screen.

Screen 3

Budget: This risk factor considers materiality of the base budget. Initiatives utilizing a

significant percentage of a department’s resources may be evaluated periodically to

confirm value for money (effectiveness, efficiency and economy).10 Helpful documents

may include: Budget Decisions, Estimates. The budget criterion is unique as there are

two screens to be completed: Budget Value and Gross Expenditure Budget. The

screen with the highest risk (1-5) must be the only screen with the “top box” checked.

10 ESB led a group of Policy Directors and PGP staff through a multiple criterion evaluation process to assign weights to the 7 factors. As a result of this process, the Budget factor is not statistically significant. Departments may choose to include initiatives which use significant resources in their Plan even if assigned a low risk assessment using this tool.

In this example, the budget value was 1. Before hitting Next, click on gross expenditure budget and complete the following screen.

Screen 4

In this example the Gross Expenditure Budget is a 3; while the Budget Value was a 1.

As the Gross Expenditure response is higher, we leave the check mark for Gross

Expenditure Budget and now hit Next (Had the Budget Value been 4 or 5, then we

would have checked the Budget Value box and hit Next). If this is lower and you have

to go back to the Budget Value screen, the previous check is removed and you will

need to reenter it.

Once you have complete the applicable Budget screen, hit Next and this will bring you

to the following screen:

Screen 5

Gross Expenditure Variance: This risk factor considers the significance of over or

under spending trends. Initiatives experiencing significant over or under spending

should be evaluated periodically to confirm adequacy and relevance (i.e. Are existing

resources adequate to address the identified population’s need). Is the initiative over-

resourced based on existing needs? Helpful documents may include: Budget

Monitoring Reports, Report on the Program Expenditures and Revenues of the

Consolidated Revenue Fund.

When your selection has been entered, click Next to bring you to the next screen.

Screen 6

Priorities: This risk factor considers how important the initiative is to achieving a

government-wide priority. Helpful documents may include: Prior year’s Departmental

Evaluation Plan, Ministerial Mandate Letters, Speech from the Throne, Public Strategy

Documents, Deputy Minister Performance Contract, Departmental Plan (Strategic,

Business, Activity), Annual Report, Operational Plan, and Work Plan.


Screen 7

Funding Renewal Commitments: This risk factor applies where evaluation may

support a request to continue funding that was approved for a fixed period or for which

there is no continuing provision in the fiscal framework. Helpful documents may include:

Fiscal Framework, Budget Decisions.


Screen 8

Public Opinion: This risk factor considers client and other stakeholder feedback about

an initiative. There may be current sensitivities, interests or pressures from clients and

other stakeholders that would warrant an evaluation (e.g. feedback from the Auditor

General, individuals, groups or MHAs). Helpful documents may include client surveys,

public consultations, Auditor General Reports, public strategy documents, departmental

correspondence such as letters to Ministers etc. If after such a review you remain unaware of client or stakeholder feedback then click on risk level 5 as there is an increased risk that there may be issues/concerns of which you are not aware.


Screen 9

Relevance: This risk factor considers the results of a department’s environmental scan

and SWOT (strengths, weaknesses, opportunities, threats) analysis. Departmental

strategic planning and or horizontal strategy development may have identified changes

in the demographics and or needs of the target population that requires a revised needs

assessment, process review and/or an effectiveness evaluation (e.g. growing wait lists

for services, lower than expected take-up). Helpful documents may include: Auditor

General Reports, Performance Monitoring and Evaluation Reports, Departmental Plan

(Strategic, Business, Activity) and Annual Report.


Screen 10

Performance Findings: This risk factor considers the findings from the performance

monitoring and or past evaluation reports. Helpful documents may include:

Performance Monitoring and Evaluation Reports, Auditor General Reports. Please note that level 5 should be selected for initiatives where there are no performance findings available. If there are no performance findings, there is increased risk that an

initiative is not performing as well as one may think.

Screen 10, performance findings, is the final of the seven risk criteria. Upon completing

Screen 10, click on the Review Selections tab.

Note: On this or any other criteria screen, you can click Back to go to an earlier screen

or if you want to go a specific criterion to review or change you can click on the

applicable top tab.

Risk Overview: Screen 11 provides an overview of the selections you have chosen for

all criteria for a particular initiative. This screen provides the options of editing your

selections or calculating your risk assessment level.

Screen 11

If you are satisfied with your risk choices, click Calculate Risk Assessment Level.

Please note you can change your calculations at a later date, by using the Revise Risk

Assessment on Screen1.

Once you select the Calculate Risk Assessment Level you will see the following screen.

Screen 12

As each additional initiative is assessed, its summary is added to your Risk Assessment

Report in accordance with its risk level, thus providing a final report of all initiatives

ranked by risk. A Risk Assessment Report, illustrated below, can be generated anytime

by clicking the View My Risk Assessment Projects tab on Screen 1 (Home Page) and

converting to Excel or Word.

Risk levels are colour coded, as illustrated below, with blue representing lowest risk and

red representing highest risk. The black markers represent the calculated risk level of

each initiative listed in your Risk Assessment Report.

Screen 12 will require you to click the EXIT button. Doing so will take you back to

Screen 1 (Home Page) where you can begin a new risk assessment on another

initiative, revise any of your previous initiatives’ risk assessments, or view all your risk

assessments.

Screen 1

To view all your completed risk assessments and produce an Excel or Word report, click

the View My Projects button. The following is an example of a screen you will see.

If you do not want to print a particular program, you can mark the box to the left of the

applicable program name and hit Remove Selected Program. Once you are satisfied

with the report you have the option to export to Excel or Word by clicking the applicable

button.

Once you are finished running reports or completing risk assessments, you can exit by

closing your browser.

Guide to Developing a Risk-Based Departmental Evaluation Plan€¦ · Web viewThe “Revised Risk...

Documents

Transcript of Guide to Developing a Risk-Based Departmental Evaluation Plan€¦ · Web viewThe “Revised Risk...