APPROACH PAPER Transport Sector Strategy: EvD Review · 2021. 1. 27. · EvD Approach Paper 4 1....

APPROACH PAPER

Transport Sector Strategy: EvD Review

November 2017

EBRD EVALUATION DEPARTMENT

EvD Approach Paper 1

The Evaluation department (EvD) at the EBRD reports directly to the Board of Directors, and is independent from the Bank’s Management. This independence ensures that EvD can perform two critical functions, reinforcing institutional accountability for the achievement of results; and, providing objective analysis and relevant findings to inform operational choices and to improve performance over time. EvD evaluates the performance of the Bank’s completed projects and programmes relative to objectives.

This approach paper has been prepared by EvD and is circulated under the authority of the Chief Evaluator. The views expressed herein do not necessarily reflect those of EBRD Management or its Board of Directors. Responsible members of the relevant Operations teams were invited to comment on it prior to internal publication. Any comments received will have been considered and incorporated at the discretion of EvD. Whilst EvD considers Management’s views in preparing its evaluations, it makes the final decisions about the content of its approach paper.

Nothing in this document shall be construed as a waiver, renunciation or modification by the EBRD of any immunities, privileges and exemptions of the EBRD accorded under the Agreement Establishing the European Bank for Reconstruction for Development, international convention or any applicable law.

Under the supervision of the Chief Evaluator, Joe Eichenberger, this Approach Paper was prepared by Saeed Ibrahim, Principal Evaluation Manager. Inputs have been provided by EvD colleagues: Tomasz Bartos, Chiara Bocci, Hiromi Sakurai and Sofia Keenan – as well as consultancy services provided by Nicholas Burke.

© European Bank for Reconstruction and Development, 2014

One Exchange Square

London EC2A 2JN

United Kingdom

Web site: www.ebrd.com

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, including photocopying and recording, without the written permission of the copyright holder. Such written permission must also be obtained before any part of this publication is stored in a retrieval system of any nature.


Contents

1. Introduction ........................................................................................................................................... 4

1.1 Objectives

1.2 Rationale for inclusion in the work programme

1.3 EBRD in the transport sector

1.4 EBRD transport operations policies

2. Transport Sector Strategy (2013) ........................................................................................................ 5

2.1 Strategic objectives

2.2 TSS management accountability and resourcing structure

2.3 TSS Results framework

3. Transport portfolio analysis .............................................................................................................. 11

3.1 Investment operations

3.2 Technical Cooperation (TC)

4. Approach and process ....................................................................................................................... 13

4.1 Evaluation overview

4.2 Developing the evaluation questions and matrix

4.3 Data collection, analysis and synthesis

5. Evaluation Questions ......................................................................................................................... 15

6. Data collection tools and sources ..................................................................................................... 16

6.1 Challenges and limitations

7. Administrative arrangements ............................................................................................................ 18

7.1 EvD team

7.2 Peer reviewers / advisory panel

7.3 Management reviewers

7.4 Timetable

7.5 Budget

Annex 1. Evaluation matrix ..................................................................................................................... 20

Annex 2. Derived transport sector strategy results framework .......................................................... 23

Annex 3. Portfolio Analysis .................................................................................................................... 26

a. Investment operations

b. Technical Cooperation (TC)

Annex 4. Relevant evaluation work........................................................................................................ 31

Annex 5. Bibliography ............................................................................................................................. 33

Annex 6. Consultations and Meetings ................................................................................................... 35


List of Figures Figure 1: EBRD Transport policy and strategic timeline, including evaluations ............................................................................... 4 Figure 2: Transport as a share of EBRD (by ABI and number of operations).................................................................................. 4 Figure 3: Infrastructure Business Group (IBG) structure as at July 2017....................................................................................... 8 Figure 4: EvD TSS Results Framework ................................................................................................................................... 10 Figure 5: TSS Evaluation Overview ......................................................................................................................................... 13

List of Tables Table 1: TSS summarised focus areas, key challenges and strategic approach ............................................................................. 5 Table 2: TSS summarised policy dialogue focus by region .......................................................................................................... 6 Table 3: TSS strategic performance indicators (SPIs) ................................................................................................................. 7 Table 4: TSS operations by framework type (Jan 2013 – May 2017) .......................................................................................... 12 Table 5: TSS operations and investment by Banking team (Jan 2013 – May 2017) ...................................................................... 26 Table 8: TSS operations and investment by status (Jan 2013 – May 2017) ................................................................................. 26 Table 9: Share of TSS operations and investment by stage and status (Jan 2013 – May 2017) ..................................................... 26 Table 10: TSS operations and investment by portfolio class (Jan 2013 – May 2017) .................................................................... 27 Table 11: TSS operations and investment by sovereign risk (Jan 2013 – May 2017) .................................................................... 27 Table 12: TSS operations and investment by instrument (Jan 2013 – May 2017) ......................................................................... 27 Table 13: TSS operations and investment by region and country (Jan 2013 – May 2017) ............................................................. 27 Table 14: TSS operations and investment by subsector (Jan 2013 – May 2017) .......................................................................... 28 Table 15: TSS operations and investment by environmental category (Jan 2013 – May 2017) ...................................................... 28 Table 16: TSS operations and investment by TI rating at approval (Jan 2013 – May 2017) ........................................................... 28 Table 17: TSS operations and investment by SEI Flag (Jan 2013 – May 2017) ............................................................................ 29 Table 19: Transport TCs by year (Jan 2013 – Dec 2016)........................................................................................................... 29 Table 20: Transport TCs by country (Jan 2013 – Dec 2016) ...................................................................................................... 29 Table 21: Transport TCs by transaction relation (Jan 2013 – Dec 2016) ..................................................................................... 29 Table 22: Transport TCs by donor (Jan 2013 – Dec 2016)......................................................................................................... 30 Table 23: Transport TCs by subsector (Jan 2013 – Dec 2016) ................................................................................................... 30

List of Abbreviations

DFF Direct Financing Facility

EBRD European Bank for Reconstruction and Development

EPG Economics Policy and Governance

EvD Evaluation Department

FOPC Financial and Operations Policies Committee

GET Green Economy Transition

IBG Infrastructure Business Group

IFI International Financial Institution

INF RCA Infrastructure Russia & Central Asia team within IBG

PPP Public Private Partnership

SEI Sustainable Energy Initiative

SOE State Owned Enterprise

SPI Strategic Performance Indicator

TC Technical Cooperation

TOP Transport Operations Policy

TRA Transport team within IBG

TSS Transport Sector Strategy (2013)


1. Introduction

1.1 Objectives

This Approach Paper sets out the focus, scope and methodological approach proposed by the EBRD’s Evaluation

Department (EvD) for its review of the EBRD Transport Sector Strategy (TSS).1 The general objectives of the evaluation are:

To provide the EBRD Board and Management with an independent, evidence-based assessment of the Bank’s

current TSS;

To identify findings and where appropriate make recommendations useful to Management and the Board during

the preparation of the succeeding strategy due by the end of 2018.

1.2 Rationale for inclusion in the work programme

The TSS was intended to apply for five years and be updated in 2018.2 EvD’s 2017 Work Programme3 committed to an

evaluation of the TSS to “encompass the operationalisation of strategic priorities, assess evidence of results relative to

objectives, and review issues related to execution and delivery of the strategy.”

Figure 1: EBRD Transport policy and strategic timeline, including evaluations

Source: EvD

This will be EvD’s second strategic review of the Bank’s approach to the transport sector. The first was conducted in 2011

as an evaluation of the Bank’s Transport Operational Policies (TOPs) 1992, 1997 and 2005 – and its analysis, findings and

recommendations were used to inform the preparation of the current TSS (Figure 1).

1.3 EBRD in the transport sector

To place the transport sector within the internal EBRD context:

the sector accounted for an average of 6% of the Bank’s operations for the last six years by number of operations

(Figure 2);

by volume it was much larger - at between 12-13% of the Bank’s annual Bank investments.

Figure 2: Transport as a share of EBRD (by ABI and number of operations)

Source: OSP – Business Performance Navigator Notes: ABI – annual Bank investment

1 As approved by the EBRD Board of Directors on 16 October 2013 2 Section 4.6 of the TSS: “the SPIs will be first reviewed five years after this document is approved by the EBRD’s Board, and a final evaluation will take place during the process of preparation of the next Transport Strategy.” 3 Evaluation Department: Work Programme 2017-19 and Budget 2017, as approved on 6 December 2016

1992 TOP

1997 TOP

2005 TOP

2011 EvD eval of 3

TOPs

2013 TSS

2017 EvD

eval of 2013 TSS

2018 TSS

6% 5% 7% 6% 7% 7% 6%

0%

20%

2010 2011 2012 2013 2014 2015 2016

As % of EBRD total ABI (reported rate) Number of operations (#)

http://boldnet2/v3_docs.nsf/0/5736C574C49071528025808100514510/$FILE/BDS16241y.pdf


1.4 EBRD transport operations policies

As depicted in Figure 1, from the period 1992-2013, EBRD’s activities in the transport sector were governed by a series of

transport operations policies (TOP).4 Briefly:

the 1992 TOP called for the Bank to take a comprehensive view of the transport sector encompassing supply and

demand, as well as organisational and environmental aspects – ensuring optimum allocation of scarce resources

and assisting better organisation and management of transport systems.

the 1997 TOP guiding principles broadly subscribed to the same elements but was also more emphatic regarding

what it called the “trilogy of principal criteria: transition impact, additionality and sound banking”.

in the 2005 TOP the Bank inter-alia sought “to assist the process of de-monopolisation, decentralisation and

privatisation”.

As noted in the 2011 EvD study, the TOP coverage has also varied over the years on the inclusion (or exclusion) of certain

subsectors such as airlines, urban transport, shipping or manufacturing for transport (e.g. wagon, sleeper, or signalling

equipment production).

2. Transport Sector Strategy (2013)

The TSS was prepared at the start of 2013 and formally approved by the Board of Directors on the 16th October 2013.

2.1 Strategic objectives

The strategy laid out what it called a vision for the region “the achievement of safe, secure and sustainable transport

systems, which embody market principles, balance economic, environmental and social needs and are responsive to the

needs of industry and the individual.”5

The TSS identified three main ‘areas of strategic focus’ to work towards realising this vision, each in response to the

identification of key challenges and an accompanying strategic approach (summarised in Table 1).

Table 1: TSS summarised focus areas, key challenges and strategic approach

Focus Area Key Challenges Summarised strategic approach

Market-based transport transport bottlenecks,

mobilising private capital and

non-sovereign financing

promoting private ownership, financing and operation of transport infrastructure, where necessary, by engaging firstly on a non-sovereign basis financing SOEs;

supporting the creation and expansion of competitive markets for transport services;

improving the efficiency of management of public sector transport assets.

Sustainability biodiversity,

climate change mitigation and adaptation,

road safety,

economic inclusion

Promotion of low carbon transport, environmental appraisal, pollution prevention and abatement, road safety planning design and investments, economic & gender inclusion through access to employment (including the construction phases of large transport projects), access to public and other services, skills transfer and improved corporate standards and practices.

4 This sections covers the following three documents (i) Transport Operations Policy, (ii) Transport Operations Policy and (iii) Transport Operations Policy 2005 – 2008 5 Transport Sector Strategy p10


Focus Area Key Challenges Summarised strategic approach

Broadening the offer shifting boundaries of the transport sector in the EBRD region, narrowing transition gaps in traditional areas of activity

more logistics, intermodal transport, including postal services,

intercity bus and coach services, road construction and maintenance,

railway property development.

Source: BDS13-205 (F), EvD

The TSS also elaborated its operational approach - how the strategy would be delivered using the Bank’s tools and

instruments. Investments were expected to observe the following approach:

retaining sovereign lending would remain as a platform whilst aiming to increase the number of private projects

as a proportion of the transport portfolio;

mobilising private capital and commercial funding wherever possible to relieve pressure on public budgets;

mobilising co-financing partners, and enhance its cooperation with other international financial institutions

(IFIs) and multilateral institutions including the EU;

promoting regional solutions and regional integration by financing key transport corridors across the region;

mobilising technical co-operation (TC) funds to support the Bank's policy dialogue and reform objectives in the

transport sector;

leveraging local knowledge, by combining sector knowledge with a deep understanding of the country and

reform context provided by resident office bankers.

The policy dialogue approach would be focused by region:

Table 2: TSS summarised policy dialogue focus by region

Region Policy dialogue focus

CEB Energy efficiency investments, financial sustainability of state railways and rolling out of PPP structures.

SEE State railway restructuring and rail regulatory development, sustainable transport policies, institutional strengthening of road agencies, development of landlord ports, and the framework for PPPs particularly for roads, airports and ports.

Turkey Corporate governance of SOEs and application of international best practice for project finance transactions, particularly PPPs, as well restructuring of the railway sector.

EEC Institutional reform, commercialisation, restructuring and financial sustainability of SOEs, regulatory development, sustainable transport policies, framework for PPPs

Russia Stronger regulatory framework; cost-based non-discriminatory tariff regimes and equal access to infrastructure for all market participants; implementation of PPP projects.

Kazakhstan Rail restructuring, autonomous SOEs in road sector, developing the framework for PPPs and adopting best practice in the development of PPP projects.

Central Asia ETC

Separation of policy from management functions, infra from ops, increase state funding for road maintenance, SOE commercialisation and institutional strengthening.


SEMED Commercialisation and restructuring of SOEs as well as development of the framework for PPPs

Source: EvD Notes: ETC – early transition countries, for other region definitions see Table 11

A distinctive feature of the TSS among the Bank’s then existing sector strategies was the inclusion of a series of strategic

performance indicators (SPIs). These were included to “support the evaluation of the achievements of the Bank in

delivering the strategy” by providing a quantifiable means of assessing whether certain strategic objectives have been met

(summarised in Table 3).

Table 3: TSS strategic performance indicators (SPIs)

Strategic Focus Area Strategic Performance Indicator

Commercial Orientation

At least 60 per cent of projects structured on a private or non-sovereign basis (calculated as an average of cumulative project numbers over a five year period.)

Sustainability Reduce CO2 emissions by a doubling of the Transport Sustainable Energy Initiative (SEI) contribution to 25% of Transport annual business volume over the next five year period (2013-2018).

Policy Dialogue Engagement

By supporting sector reform and restructuring enable the transformation of 3-5 sovereign clients into commercially-oriented entities (or part of their operations into commercially-oriented ring-fenced activities), which can raise finance on a non-sovereign basis or contract-out operations to the private sector.

Road Safety Within five years all public sector road projects are subject to a road safety assessment that identifies the risks to be reduced and at least 50 per cent of such projects to include specific road safety components or initiatives to enhance the impact of the Bank’s project on improving road safety in its COOs.

Source: BDS13-205 (F)

EvD notes that the majority of the SPIs were at the output level, and internally focused (i.e. related to features to do with the

preparation and structure of Bank projects – assessments carried out, SEI flag, sovereign classification etc.). The strategy

stated the “SPIs [are] unavoidably skewed to those objectives that are not only measurable but that have region-wide

relevance”.

This effort to provide a more “quantifiable means of assessing whether certain strategic objectives have been met” can be

viewed as part of wider changes within the Bank at that time, culminating in the 2014 EBRD paper that sought to update the

Bank’s overall results architecture to tell a more compelling transition story.6 In that paper it was established that the country

strategy framework would become the main repository for the Bank’s results in terms of outcomes and impacts, and that

“sector strategies and initiatives will have performance monitoring frameworks that set clear objectives and track

performance through key output level indicators.”

2.2 TSS management accountability and resourcing structure

The TSS was presented for Board approval by the Managing Director of the Infrastructure Business Group (IBG) within

Banking (Figure 3: Infrastructure Business Group (IBG) structure as at July 2017Figure 3). The IBG is comprised of three teams,

and the Bank’s transport projects are spread across two of those teams. There are two ‘sectoral’ teams, Municipal and

Environmental Infrastructure (MEI) and Transport (TRA) respectively, and one regional team, Infrastructure Russia Central

6 The Architecture of Transition Impact Results Frameworks in the Bank in 2014

http://boldnet2/v3_docs.nsf/0/F2D144E83142266180257D5C0050010C/$FILE/CSFO1427.pdf


Asia (INF RCA). This regional team originates, develops, monitors and reports on all municipal and transport projects in that

region. The review will seek to understand the TSS management structure in terms of the development, ownership,

monitoring and execution of the sector strategy.

Figure 3: Infrastructure Business Group (IBG) structure as at July 2017

Director, Infrastructure, Russia & Central Asia

Director, Transport Director, MEI

Managing Director, Infrastructure

Transporti. sector strategy, ii. projects and iii. portfolio

2.3 TSS Results framework

EvD develops results frameworks to show a series of assumptions and causal relationships leading to the results identified

in the operation or strategy under assessment. The aim is to help:

show how EBRD’s support (inputs) contributes to outputs, outcomes and impacts, and

help to structure the review – in particular the identification of the evaluation questions.

Results Frameworks were not routinely prepared in EBRD in the past – and this was the case with the 2013 TSS.

EvD therefore retrofitted a results framework for the TSS, wherein all explicit and implicit objectives set out in the TSS –

were mapped into a structure of outputs, outcomes and impact to create a hierarchy of results (Annex 2).7

Using this as an analytical foundation, EvD adapted this results framework (Figure 4) to provide a more simplified and

logical flow by:

including lines of logical progression for each major result at the different levels – which were often implicit in the

TSS and to limit disjointedness in the results;

aggregating the TSS outputs into three groups (per the type of intervention supported by EBRD - physical project,

policy, or sustainability initiative) – for greater clarity;

differentiating the outcomes between those that could be expected to be observed relatively early (perhaps on an

intervention-by-intervention basis), and those that could only be expected to arise later (and probably as a

consequence of several interventions combined).

This EvD results framework identifies six main outcomes for the TSS, namely:

The alleviation of transport bottlenecks

Deepening private sector participation

7 This results framework (to the extent possible) takes the contents of the TSS exactly and exhaustively. The only external judgment contained - are on which level each stated objective should sit. This derived results framework is rendered to more fully reveal the rationale and objectives of the TSS as stated (it also contains the four SPIs identified in the TSS).


Expansion of competitive markets for transport services

Improved efficiency in the management of public sector transport assets

Improved transport policy and regulatory environment

Improved transport sustainability, safety and security


Figure 4: EvD TSS Results Framework


3. Transport portfolio analysis

3.1 Investment operations

The population taken into consideration for the purpose of this review is the portfolio of the transport operations

managed by the two aforementioned teams: TRA and INF RCA.

The timeframe under consideration is all projects approved and signed between January 2013 when the TSS was first

drafted - and signed as at end of May 2017. The above approach has been discussed and validated by Management.8

Using this agreed criteria - resulted in the identification of 70 operations to be considered as the population for this TSS

review. The data and sources underlying this portfolio analysis are provided in Annex 3 - an overview is provided below:

There have been 70 approved and signed transport operations during the TSS period to date for net

cumulative investment of €3.2 billion;

With 51 of the approved projects, the majority of the portfolio is managed by the TRA team in IBG, and 19

transport projects have been approved by the INF RCA team (Table 5);

The project vintage is relatively young, with fully 64 out of the 70 projects approved still active. This

translates to 91% active by number of approvals, and by volume of investments this increases to 97% active

and only 3% complete (Table 6).

Breaking the active operations down further, by the stage of the investment, almost a third of approvals are

currently at the signature stage and have not begun disbursing, a further 21% have begun disbursing but do

not yet have any repayments, and a much larger 41% have started repaying (Table 7);

59% is in the private sector by number of operations (41) and 41% in the state sector (29). However by

volume of investment the share of these private sector approvals falls to 40% with the balance of 60% in the

state sector (Table 8).

48 out of 70 operations were approved on a non-sovereign basis (69%). These 48 operations accounted

for a business volume of €1.53 billion which translates into a smaller share (48%) by volume of financing. On

average sovereign projects undertaken during the TSS period have been over twice the size of non-sovereign

projects during the TSS (Table 9 ).

By approvals, the Bank’s transport projects have been almost entirely (90%) financed through debt

instruments, with equity accounting for only 7% of operations. The debt share increases to 95% by volume

of financing. This stands to reason, given the historic shares of sovereign loans and the types of transport

clients, e.g. SOEs (Table 10).

By number of operations Central Asia has seen the highest level of activity during the TSS period with 16

approvals (23%), followed by South Eastern Europe with 15 approvals (21%) and Eastern Europe and

Caucasus with 12 (17%). By volume of investment South Eastern Europe received 24% followed jointly by

Central Europe and the Baltics and Central Asia with 16% respectively (Table 11).

Kazakhstan has far and away been the biggest beneficiary country both in terms of volume and by number of

operations (14, .which counts 20% of the total), followed by Turkey (8), Bosnia & Herzegovina, FYR

Macedonia, Poland and Ukraine all with 4 approvals respectively. By amount of financing, Kazakhstan has

benefitted from €521 million (16% of the total), followed by Turkey with €343 million and FYR Macedonia €315

million (Table 11).

8 Though there are ongoing discussions between EvD and Management around special cases where the application of the criteria is not straightforward. Once resolved these will be fully clarified in the evaluation report itself.


Four subsectors together account for roughly three quarters of approvals and volume of financing. 17 road

construction projects accounted for 24% of approvals and 36% of the investment, 16 intermodal transport

projects accounted for 23% of approvals and 13% of investment, 14 rail transport operations amounted to

20% of approvals and 18% of investment and seven ports and harbours operations accounted for both 10%

of approvals and investment (Table 12).

In terms of environmental and social impact, the Bank’s transport projects under the TSS have predominantly

been classified as B9 (84% of approvals and 67% by volume). However there were seven operations

classified as A (could potentially result in significant adverse environmental and social impacts) which

amounted to 25% of volume (Table 13).

In terms of transition impact, at approval, the majority of projects (79%) were rated at various levels of

Good (55 out of 70 approvals) (Table 14).

35 of the 70 projects were flagged as containing SEI/GET components, giving a roughly 50% SEI/GET

Transport contribution by number of operations. Moreover, for the 35 operations a total amount of €1.03 billion

of SEI finance was flagged (Table 15).

40 out of the 70 projects had external co-financing. 9 of the 40 projects were also joint operations with other

IFIs - who provided around €1.6 billion of investment.

Of note is that around one in five operations have been approved by the Small Business Investment Committee

as sub-operations under larger frameworks (Table 4 most frequently, under the Direct Financing Framework (DFF).

There are 13 sub operation approvals which account for 19% of approvals but understandably for a much lower

proportion of volume (3%). The DFF itself was established as part of the Small Business Initiative in late 2015, wherein 4

pre-existing facilities devoted to direct investments were consolidated.

Table 4: TSS operations by framework type (Jan 2013 – May 2017)

Framework type Number of ops % Investment (€) %

Stand alone 57 81% 3,107,078,894 97%

Sub operation 13 19% 80,746,522 3%

Source: DataWarehouse

3.2 Technical Cooperation (TC)

One of the main elements of the TSS operational approach was mobilising TC funds to support the Bank's policy

dialogue and reform objectives in the transport sector. Based on data provided by the Infrastructure TC Unit, during the

TSS period:10

the Bank administered 111 TCs for the transport sector, for a total of €34.4 million (Table 16)

the vast majority by number (102) and by amount (€31.2 million) have been transaction related, with the

balance stand-alone TCs (Table 18).

by number of TCs, FYR Macedonia has been the biggest beneficiary of transport TCs by some distance

with 16 followed by Bosnia and Herzegovina and Kazakhstan with 10 respectively. The remainder of the top 10

recipients can be seen in Table 17, all receiving at least six separate TC assignments.11

in terms of donor source, the bulk of the support has come from the EBRD itself through its Shareholder

Special Fund (SSF). Fully 46% (51 of the 111) transport TC operations has been funded by the SSF.

Similarly, in terms of volume, 35% of the TC amount (€11, 9 million) has come from the SSF. Other major

donors have been the Western Balkans Investment Framework (€9.4 million for eight TCs), the Early

9 Environmental and social impacts are an important feature of the Bank's mandate, and as per the Bank’s environmental and social policy - EBRD categorises projects at appraisal to reflect the level of potential environmental and social impacts and issues (with Category A being those projects associated with the highest E&S impact) 10 The data provided by the Infrastructure TC Unit covers the period 2013-16 11 Morocco, Moldova, Montenegro, Croatia, Albania, Kosovo, Russia and Egypt


Transition Countries Fund (2.1 million for three TCs) and the Central European Initiative (1.7 million for seven

TCs). A complete list of the donors is available in Table 19.

by subsector, the majority of the TCs have supported rail operations (€21.1 million for 56 TCs) followed by

road operations €5.4 million for 27 TCs and ports €3.1 million for 11 TCs (Table 20).

4. Approach and process

4.1 Evaluation overview

This review will follow selected aspects of the standard OECD DAC evaluation principles (in particular relevance,

effectiveness and efficiency).

For a strategic review these can be interpreted as relating to the contextual background and assumptions that underpin

the strategy, the process of implementing the strategy and measuring its results. EvD has depicted how the evaluation

questions it will explore in this review; correspond to these elements of the strategic cycle (formulation, execution

and realisation) in an evaluation overview (Figure 5). This overview also illustrates how the results framework fits into

the wider review and also aims to capture the evaluation process and route to potential findings and recommendations.

Figure 5: TSS Evaluation Overview

4.2 Developing the evaluation questions and matrix

For the identification of the main issues of interest for the review it was necessary to gain an understanding of:

the context within which the TSS was approved,

the strategic objectives of the TSS and

the activities and projects that correspond to the implementation of the TSS.

The context was explored through analysis of internal EBRD documents and interviews with key EBRD stakeholders –

for documents consulted see Annex 5, and for key stakeholders consulted see Annex 6.


The implementation of the TSS has been documented through the initial analysis of the transport portfolio of operations

and TCs – (summary is presented in section 0 and detailed in Annex 3). The portfolio analysis also facilitated the

identification of potential countries to be visited and areas of interest (section 6).

The objectives and key elements of the TSS, described in section 2.2, have been represented in the EvD results

framework (Figure 4).

The identified TSS impact(s) should signify contribution to (or progress towards) the Bank’s strategic objectives in the

sector – whilst recognising that impacts are generally more difficult to define at the sector level.

The Bank’s sector vision presents its intended impacts as: “the achievement of safe, secure and sustainable transport

systems, which embody market principles, balance economic, environmental and social needs and are responsive to the

needs of industry and the individual.”

From an evaluation perspective, the sector vision as stated does not include or imply elements for meaningful or

measurable assessment. For this reason, EvD’s assessment of the effectiveness of the TSS will focus on

assessing the extent of achievement thus far of the identified outputs, immediate and intermediate outcomes.

EvD will explore impact issues in the following way:

in light of the updated transition concept adopted in October 2016 that “a sustainable market economy is

one that is competitive, well-governed, green, inclusive, resilient and integrated"

EvD will attempt to map the current TSS population to the six new transition qualities framework, noting;

the potential of TSS operations to contribute to transition qualities; and

where a new strategy might accomplish better alignment.

This analysis will not be used as an effectiveness measure, and will be considered under the question of the TSS’s

continued relevance to the EBRD institutional context.

4.3 Data collection, analysis and synthesis

Based on these inputs the evaluation matrix of evaluation questions and sub-questions (judgment criteria and indicators)

was developed (see section 5 and Annex 1). The evaluation matrix is intended to serve as the analytical framework to

guide and structure data collection phase - via the tools and methods for described in section 6. While no significant

revisions of the matrix are foreseen, the evaluation matrix might be slightly amended during the subsequent phases of

the review to reflect new inputs substantive for achieving the objective of the review. The data collection will be followed

by a synthesis phase - devoted to systematically constructing answers to the evaluation questions and formulating

conclusions and recommendations on the basis of the data collected throughout the process.


5. Evaluation Questions

The main evaluation questions and their underlying rationale are briefly outlined below. The full evaluation matrix,

which will be used to guide the collection and structuring of data, is presented in Annex 1.

The evaluation team has identified three evaluation questions to guide this review of the TSS which broadly correspond

to the OECD/DAC criteria of relevance, efficiency and effectiveness.

EQ1: Was the TSS relevant to the requirements of the COOs and the institutional context of EBRD?

Did the TSS: (i) identify market developments, challenges and transition gaps in the transport sector and set ‘focus

areas’ and operational approaches to directly address them; (ii) distinguish where EBRD could be additional in the

sector; (iii) incorporate aspects of best practice in its design; (iv)align with the evolving EBRD strategic context.

EQ2: What results have emerged from the TSS implementation so far?

This evaluation question is directly related to actual EBRD operations approved, signed and implemented since the TSS

was drafted in 2013. Bearing in mind this is an evaluation of an on-going strategy, the evaluation team will seek to carry

out a preliminary assessment of the progress that the Bank has made in reaching its intended results in the transport

sector through its investment operations, technical cooperation and policy dialogue activities. As explicitly indicated in

the results frameworks, the TSS results can be summarised by six expected outcomes. The evaluation team will collect

data to gauge progress toward the emerging results related to the six outcomes. The evaluation team will also attempt tp

update the four SPIs identified in the TSS.

EQ3: How efficiently has the Bank implemented the TSS and its operations?

The evaluation team will be focusing on the extent to which the strategy has guided the Bank’s activities (broadly

defined) and the efficiency of those activities. Answering this evaluation question could include an assessment of: (i)

complementarity and alignment of subsequent country strategies and projects with the objectives of the TSS; (ii)

mobilisation of the targeted resources to support the achievement of the TSS objectives; (iii) coordination of activities

with other IFIs and partners; (iv) the organisational arrangements put in place to execute the TSS; and (v) the

implementation and financial performance of TSS activities.

Based on an analysis of the above, the review will to the extent possible - identify findings, conclusions and

recommendations to be considered during the preparation of the next iteration of the TSS.


6. Data collection tools and sources

A series of tools will be used for collecting, structuring, processing and analysing data. Some have already been used in

the preparation of this Approach Paper (such as preliminary review of documents, initial interviews, and portfolio

analysis). This section presents the tools and collection methods foreseen for the use in all the phases of the review.

EvD will mix the collection methods as fits the purpose of the specific judgement criteria identified to respond to the three

evaluation questions (see the evaluation matrix in Annex 1).

Interviews

First exploratory interviews with key stakeholders have already been conducted in the course of the preparation of this

Approach paper (Annex 6). Further interviews, structured in line with the evaluation matrix, will be conducted both in

EBRD HQ and in the countries selected for field visits. Some interviews may be conducted via phone or

videoconferencing as necessary. The team will seek to gather the views in particular of the following key stakeholders

(not exhaustive):

IBG staff in HQ and ROs

Heads of ROs

Representatives of other relevant EBRD teams, including: E2C2, EPG/OCE, ESD, Vice Presidency Policy,

Donor Co-financing, Legal Transition Team, Internal Audit, etc.

Representatives of EBRD shareholders and donors

IBG clients

Relevant counterparts/stakeholders at country level (including local and national authorities, regulatory

agencies, sector associations, civil society organisations, etc.)

Representatives of IFIs, bilateral donors, etc.

Country visits

Operationally, the TSS was meant to provide the parameters based on which the EBRD would set priorities at country

level. The evaluation team intends to visit selected countries of operations to complement the desk work. The objective

is to gather information to feed into the answers to evaluation questions at the country level, and to explore the different

contextual determinants of TSS implementation and success. The rationale is that the Bank is predominantly seeking to

address transition challenges at country level. It is important to emphasise that the country visits will not be aimed to be

stand-alone evaluations of EBRD country strategies.

EvD will agree with Management on two countries to visit from a short-list of four, namely: Bosnia & Herzegovina, FYR

Macedonia, Kazakhstan and Turkey. Selection criteria included:

The number of projects (and volume) signed: critical mass is essential to achieve the outcomes pursued by

the Bank through the TSS and the projects and other activities that deliver these;

The existence of substantial technical cooperation and policy dialogue-related interventions;

Transition impact ratings at approval: a higher proportion of projects rated good or higher denotes a high

level of potential impact in the country and higher likelihood to achieve one or more of the expected outcomes

of the TSS;

Countries belonging to different regions characterised by different transition and generally early and

intermediate transition stages or countries from the different banking team responsibilities.


EvD will conduct a preliminary assessment of the portfolio for contribution to each of the six major outcomes – as

additional criterion for country visit selection.

Logistical arrangements will be discussed and coordinated with RO teams in advance, and success of the missions will

depend to a large degree on the cooperation of the IBG colleagues in HQ and EBRD Resident Offices staff, especially in

terms of provision of contact details for local clients and other stakeholders.

Specific themes of interest

To complement data collection and structuring along strict geographical lines, other aspects will be investigated. Given

the importance given by the TSS to themes (see section 1) the evaluation team also consider the Bank interventions in

the sector through cross-cutting or thematic lens. For instance, a preliminary topic of interest will be the use of the so-

called Direct Finance Facility.

Other topics for thematic case studies could be added to the review where they would help to answer the evaluation

questions and resources and time will allow.

Portfolio analysis

Preliminary analysis of the transport portfolio over the period January 2013-May 2017 has already contributed to the

preparation of the methodology (Annex 3) and for the preliminary selection of countries for field visit. During the review, it

might be further developed as needed to contribute to further data collection. The same approach applies to the

population of grants available in the period under consideration. Data analysis will also be used for selecting

stakeholders to be interviewed, in collaboration with the transport and country teams.

Documentary analysis

Similarly, preliminary reading of key TSS-related and contextual documents has already been carried out for the purpose

of preparation of this Approach Paper (Annex 5). Further documents will be collected and consulted to complement

information relevant to the issues outlined in the evaluation matrix. This will include but not be limited to the following

documents:

EBRD Board-approved documents relating to activities in the transport sector, including strategies, policies,

programme documents, technical cooperation and non-technical cooperation grants; minutes of EBRD Board

of Directors meeting and its committees discussions on transport related matters; Board information sessions;

etc.

EBRD Country Strategies approved after October 2013, etc.

IBG projects-level documents, including TC documents and available reports, etc.

Contextual documents and statistics relevant to IBG countries of operations, including EBRD Assessment of

Transition Challenges as included in the annual Transition Reports, national development strategies, relevant

documents of OECD, DFIs, IEA and other international organisations, etc.

Previous relevant evaluation work, some of which has been initially reviewed in Annex 4:

6.1 Challenges and limitations

EBRD strategic planning context

The TSS was approved in October 2013. Since then the following EBRD strategic planning developments have occurred

which will need to be taken into account during the review of the TSS:

The TSS was approved and interlinked with the EBRD SEI – specifically one of the SPIs had an SEI target.

Since then the SEI was superseded by the SRI in 2014, and eventually in 2015 the EBRD launched the Green

Economy Transition (GET).


Transition Results Management is undergoing a major restructuring through Project Christopher touching on

the broad results architecture in the Bank – project and Country Strategy level.

Immaturity of the TSS portfolio

Evaluating the effectiveness of the strategy will be a challenge. Most of projects are at early stages of implementation -

91% of the TSS portfolio is active by number of approvals, and by volume of investments this increases to 97%. This will

mean that, at the project level, in a number of cases even outputs (physical results) will not yet be due, let alone

outcomes and impacts. Similarly, in terms of the assessment of outcomes the assessment will be focused on progresses

towards outcomes rather than actual achievement.

Results contribution

The evaluation team will seek to identify and assess the contribution of EBRD to observed results, mainly through

qualitative analysis. The source of information may come from the EBRD itself, and from direct interviews carried out as

part of the review with relevant stakeholders. In addition, views and opinions will be collected as a part of direct

interviews in the countries visited.

Availability and quality of available information

Experience from past EvD Special Studies shows that the necessary documentation may not always be fully available to

the evaluation team. In addition, Special Studies of large programmes rely to some extent on available monitoring and

evaluation data and their quality and comprehensiveness. Lastly, the availability of EBRD current clients for interviews

and their willingness to share information with the evaluation team may affect the data collection, as may the availability

of other counterparts and stakeholders.

Staff turnover

Related to the previous issue is the challenge of staff turnover and institutional memory. It is a common occurrence that

key staff changes both internally in EBRD, and externally within local stakeholders and counterparts. This may hinder the

collection of data of older date, as institutional memory may not be preserved.

7. Administrative arrangements

7.1 EvD team

The evaluation team is composed of Saeed Ibrahim, EvD Principal Evaluation Manager. As appropriate, specific

contributions may be made by other EvD colleagues, in particular for specific data collection in countries of operations if

efficient.

The evaluation team will benefit from the services provided by a consultancy team – to be contracted based on the

needs of this Evaluation on specific aspects.

7.2 Peer reviewers / advisory panel

Mr Barry Kolodkin, EvD Deputy Chief Evaluator, will be the internal peer reviewer of this study.

The evaluation team will select one external peer reviewer. The expert will provide an independent review of the final

draft of the study with respect to its quality, in particular soundness of the analysis, appropriate evidence base and

presentation of findings, links between findings, conclusions and recommendations, etc.

7.3 Management reviewers

The draft Approach Paper has been circulated to the designated counterparts in Management as per the Operations

Manual 8.5, namely to the two Focal Points (Director Strategy & Policy Coordination in VP Policy, and Managing Director


Portfolio Business Group) who co-ordinated with the relevant departments/teams for circulation of the draft Approach

Paper and shared consolidated comments with EvD. Management Focal Points and Reviewers for the study include:

Management Focal Points:

Managing Director, Portfolio – Kanako Sekine, [email protected]

Director, Country Strategy Coordination & Results Management – Christoph Denk, [email protected]

Management Reviewers:

Managing Director, Infrastructure – Jean-Patrick Marquet, [email protected]

Director, Transport – Sue Barrett, [email protected]

Director, Infrastructure, Russia and Central Asia– Ekaterina Miroshnik, [email protected]

This list may be added to during the course of the review.

7.4 Timetable

Milestone Date

Study starts April 2017

Draft Approach paper circulated to Management August 2017

Approach paper approved August2017

Field visits October & November 2017

Draft circulated to internal peer reviewers January 2018

Draft cleared by CE for circulation to external peer reviewers January 2018

Draft cleared by CE for Management Comments February 2018

Final approved by Chief Evaluator March 2018

Final distribution to Board April 2018

7.5 Budget

This study will require resources for EvD staff, travel costs and external consultant for peer review, all within EvD

approved budget for 2017-18. A sector expert for reform benchmarking against international standards might be hired on

a need basis.

mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


Annex 1. Evaluation matrix

Evaluation questions Judgement criteria Indicators Sources of Data

1. Was the TSS relevant to the requirements of the COOs and the institutional context of EBRD?

1.1 TSS reflected market developments, challenges and transition gaps identified in the transport sector and set ‘focus areas’ and operational approaches to directly address them

Existence of assessments of sector developments, challenges and TI gaps

Coherence between sector/gaps analysis and focus areas

Operational approach tailored to address specific challenges

TSS, interviews Preparatory analysis to TSS OCE/EPG documents External literature

1.2 TSS was aligned to the (evolving) EBRD strategic and institutional context

TSS alignment with strategic documents at the time e.g. CRR4 and SEI

Continued alignment of TSS with updated strategic documents e.g. SCF, SRI and GET

Potential contribution of the TSS to newer strategic initiatives e.g. Inclusion, Gender & LC2

Potential TSS contribution under the evolving transition concept

TSS Other EBRD strategic planning documents e.g. CRR4, SCF, SIP, SEI 3, SRI GET, TI concept etc., interviews

1.3 TSS included an identification of where EBRD could be additional in the sector

TSS includes an ex-ante assessment on how EBRD’s interventions are likely to be additional through its: (i)Terms, (ii) Attributes and (iii) Conditions

TSS, interviews Preparatory analysis to TSS

1.4 Design of the TSS reflected best practice:

set clear and complete objectives?

included an adequate results framework for achieving its objectives

included sufficient reflection of past experience (and used it to shape design)

identified resources required for achieving its objectives?

TSS set clear priorities and objectives, measurable targets and monitoring indicators

Existence of explicit or implicit architecture of results within the TSS

Reference to and use of findings and lessons from the previous transport policy and other evaluations

Identification of level resources required for achieving objectives

TSS and underlying documents interviews

2. What results have emerged from the TSS implementation so far?

2.1 Progress towards the alleviation of transport bottlenecks? Contribution of TSS interventions to:

Increased capacity of roads, railways, airports and ports

Improved access of businesses and consumers to markets and services

Transport infrastructure better integrated with European transport networks

Project documentation (investments, TCs and policy dialogue) External literature, interviews International databases

2.2 Progress towards deepening private sector participation in the transport sector

Contribution of TSS interventions to:

increased number of private management and operational contracts

increased number of private investment in construction and mgmt. of infrastructure (e.g. BOT/PPP)

increased number of full privatisations – (demonstrated through Bank’s participation in IPOs or private placements)

(EBRD level indicators)

transformation of 3-5 sovereign clients into commercially-oriented entities (SPI 1)

increased number of non-sovereign loans to sustainable SOEs

greater selectivity in sovereign operations

Project documentation (investments, TCs and policy dialogue) Internal Bank databases External literature, interviews International databases

2.3 Progress towards expanded competitive markets for transport services?


expanded market and increased competition in road construction and maintenance

new, innovative products developed (where markets for services already exist)



improved corporate governance through the introduction of international standards

freight: state operators commercialised and corporatized, with their operations streamlined, and the market-based tariffs introduced (where markets yet to be liberalised)

2.4 Progress towards improved efficiency in the management of public sector transport assets?


increased separation of transport infrastructure policy and management functions (demonstrated through the creation of autonomous SOEs outside of direct government control e.g. independent road agencies)

improved corporate governance and business practices in SOEs (demonstrated by the ability to attract funding on commercial terms)

improved transparency and accountability in the management of transport infra

improved financial sustainability of maintenance and ops, cost recovery & contractualisation


2.5 Progress towards improved transport sustainability, safety and security?


energy consumption in transport services reduced

CO2 emissions in transport sector reduced (SPI)

safety in the transport sector improved (through improved road safety throughout EBRD’s ops)

greater inclusion in the transport sector (reduced disparities in access to employment, particularly for women and vulnerable groups or those living in remote rural or underserved urban areas)

reduced number of road and rail accidents;

improved access to employment, including local workforce participation in the construction phases of large transport projects, access to public and other services, skills transfer


2.6 Progress towards an improved transport policy and regulatory environment?


legislative and regulatory framework for PPPs

vertical separation where infrastructure and operations remain within a single entity (mainly in the rail sector)

introduction of rail access charging regimes, competitive tariff structures and concession law

institutionally strengthened regulatory agencies

increased contractualisation (introduction of contracts between the government and SOEs responsible for providing transport infrastructure and services)

improved cost recovery through reformed tariff structures and streamlined operations (user pays’ principles in road whilst ensuring that affordability constraints are taken into account)

increased private sector participation in commercial transport services

introduction of sustainable transport policies and best practice



3. How efficiently has the Bank implemented the TSS?

3.1 TSS has been used to (demonstrably) guide subsequently approved country strategies and projects?

EBRD country strategies incorporate sections about the transport sector with results and indicators that aligned with TSS objectives and SPIs

Project documents refer to which strategic focus area and SPI that they aim to contribute

28 Country Strategies approved after October 2013, interviews Projects documents for the 70 TSS projects

3.2 The Bank was able to mobilise the targeted resources to support the achievement of the TSS objectives

Extent of mobilisation from private sector

Extent of mobilisation from donors for TCs

Extent of mobilisation from IFI’s for cofinancing

Data and information from IBG and OS&P teams, interviews Infrastructure TC Unit

3.3 TSS activities have been coordinated with other IFIs and partners

Existence of mapping of other IFI/donors on-going or planned activities in the sector

Demonstrations of complementarity of actions and activities at the country level

Examples of other co-ordination work e.g. on Joint MDB initiatives etc.

TSS, interviews Preparatory analysis to TSS Data and information from IBG team

3.4 EBRD provided sufficient organisational arrangements to execute the TSS

Management responsibilities and incentives aligned with TSS objectives

Adequacy of staffing and budgeting

Monitoring and reporting arrangements

Accountability mechanisms towards the Board

Structured mechanism of learning and knowledge management

IBG scorecard, interviews Other internal EBRD documents

3.5 TSS activities have been financially and resource efficient Financial performance of projects/clients

Implementation efficiency of projects/TCs

Investment profitability of the TSS portfolio

IBG scorecards, interviews Internal Bank databases Project monitoring documents


Annex 2. Derived transport sector strategy results framework Assumptions Inputs / Assets Activities Outputs: Outcomes: Impacts

Governments are willing

to introduce effective

laws and sector

regulations to establish

the regulatory

environment for private

sector participation

based on transparency

and accountability. (p27)

Other IFIs and

financiers are willing to

co-finance transport

projects (p25)

EBRD’s financing resources in

hard and local currency (4.3):

sovereign lending ( where large transition gaps exist)

senior and mezzanine corporate debt

project finance

capital market investments

guarantees

equity participation

Financing transport projects

in traditional sub-sectors - road, rail, maritime and aviation (Annex E):

in niche sub-sectors e.g. logistics and intermodal (3.3)

including both of the above with smaller clients through special facilities e.g. LEF (p43)

At least 60 per cent of projects structured on a private or non-sovereign basis (SPI 2)

Physical transport infrastructure built and/or

goods delivered

A. related to Bank’s traditional subsectors

(i) in the road sub-sector:

construction and rehabilitation of roads,

trans and pan European road corridor development

performance based contracts introduced

road safety assessments conducted and components introduced into projects

(ii) in the rail sub-sectors (Annex E):

infrastructure rehabilitation, upgrade or new construction

rolling stock renewal

financial and labour restructuring

infrastructure and rolling stock support services

energy efficiency and station development

(iii) in the maritime subsectors (Annex E):

shipping fleet modernisation

port capacity expansion

efficient port infrastructure in the region

(iv) in the aviation subsectors (Annex E):

airport infrastructure

associated services, such as aircraft maintenance

aircraft acquisition and other airline investments (selectively)

Transport bottlenecks alleviated (key challenge 3.1), including:

Capacity of roads, railways, airports and ports increased (demonstrated through reduced congestion and increase of goods and people flow through main transport hubs and networks);

Access of businesses and consumers to markets and services improved (demonstrated through the increase of inter-regional and domestic trade) (exec sum)

Transport infrastructure better integrated with European transport networks (demonstrated through increase in international trade between EBRD region and the rest of Europe)

Focus 1: Market-based transport

Private sector participation deepened( increased private ownership, financing and operation of transport infra) (3.1.2)

increased number of private management and operational contracts (p27)

increased number of private investment in construction and mgmt. of infrastructure (e.g. BOT/PPP) (p27)

increased number of full privatisations – (demonstrated through Bank’s participation in IPOs or private placements) (p27)

(EBRD level indicators)

transformation of 3-5 sovereign clients into commercially-oriented entities (SPI 1)

increased number of non-sovereign loans to sustainable SOEs (p47)

reduced number of sovereign operations (p26)

Competitive markets for transport services expanded (3.1.2)

expanded market and increased competition in road construction and maintenance (p36)

new, innovative products developed (where markets for services already exist) (p28)

improved corporate governance through the introduction of international standards (p28)

freight: the f state operators commercialised and corporatized, with their operations streamlined, and

Efficient, safe,

secure and

sustainable

transport

systems,

which embody

market

principles,

balance

economic,

environmental

and social

needs and are

responsive to

the needs of

industry and

the individual

(TSS sector

vision)

Donor resources for EBRD

TCs and policy dialogue

Mobilising resources

private capital and commercial funding e.g. through PPPs and syndication (4.4)

public co-financing partners, by enhancing its cooperation with other IFIs and EU (4.4)

& aligning reform conditionality of co-financing IFIs (p45)

TC funds to support (4.5):

the Bank’s policy dialogue and reform objectives

sustainable transport

project prep and implementation

investment grants

Governments are willing

to undertake structural

reforms to create

financially autonomous

SOEs which can support

commercial funding

(p25)

Co-financing & syndication

resources (from private sector

and IFIs mainly)

Policy dialogue and cooperation efforts to

promote transition related reforms,

environmental practices, energy efficiency and

other important policy areas, etc.:

with Govts/authorities

with national and international organisations or through participation in international initiatives e.g. multilateral development bank road safety initiative & rio+20 joint statement undertakings

stakeholder engagement either on a project-basis or through other channels

Focus 3: Broadening the Offer

Related to “niche” sub-sectors e.g. (3.3):

logistics, intermodal transport, including postal services

intercity bus and coach services and associated infrastructure such as bus terminals.

the renewal of buses with higher EE and environmental performance

private road construction and maintenance

railway property development.


Assumptions Inputs / Assets Activities Outputs: Outcomes: Impacts

the market-based tariffs introduced (where markets yet to be liberalised) (p28)

Efficiency of mgmt. of public sector transport assets improved (3.1.2)

increased separation of transport infrastructure policy and management functions (demonstrated through the creation of autonomous SOEs outside of direct government control e.g. independent road agencies) (p27)

improved corporate governance and business practices in SOEs (demonstrated by the ability to attract funding on commercial terms) (p27)

improved transparency and accountability in the management of transport infra (p27)

improved financial sustainability of maintenance and ops, cost recovery & contractualisation (p28)

Private financiers willing

to co-finance transport

infrastructure using

privatisation or

concession structures

such as PPPs

EBRD’s attributes

staff expertise - leveraging local knowledge (RO/sector) p39

influence /reputation

covenants/conditions p45

Focusing on environmental and social

sustainability

all projects designed and implemented in compliance with E&S policy (env appraisal, pollution prevention and abatement, biodiversity, resettlement and livelihood restoration, stakeholder engagement) (p32)

road safety assessments carried out (100%) and road safety components (50%) included into road sector projects (SPI 4)

economic inclusion and gender issues considered in projects supported

Policy and regulatory changes implemented,

related to (replicated in outcomes (4.2 and 4.5):

CEB: EE investments, financial sustainability of state railways and rolling out of PPP structures (including enhancing business and investment planning in port authorities in Croatia)

Turkey: corporate governance of SOEs and application of international best practice for project finance transactions, particularly PPPs, as well restructuring of the railway sector.

EEC: institutional reform, commercialisation, restructuring and financial sustainability of SOEs, regulatory development, sustainable transport policies, framework for PPPs (including road sector reform in Ukraine and Western Balkans, as well as state railway company restructuring in the Balkans)

Russia: stronger regulatory framework; cost-based non-discriminatory tariff regimes and equal access to infrastructure for all market participants; implementation of PPP projects

Kazakhstan: rail restructuring, autonomous SOEs in road sector, developing the framework for PPPs and adopting best practice in the development of PPP projects.

Central Asia ETC: separation of policy from management functions, infra from ops, increase state funding for road maintenance, SOE commercialisation, corporatisation and institutional strengthening including road sector reform.

SEMED: commercialisation and restructuring of

Transport policy and regulatory environment improved,

related to: (3.1.2 & 4.2)

legislative and regulatory framework for PPPs (end 4.2)

vertical separation where infrastructure and operations remain within a single entity (mainly in the rail sector) (p27)

introduction of rail access charging regimes, competitive tariff structures and concession law (p28)

institutionally strengthened regulatory agencies (p28)

increased contractualisation (introduction of contracts between the government and SOEs responsible for providing transport infrastructure and services) (p28)

improved cost recovery through reformed tariff structures and streamlined operations (user pays’ principles in road whilst ensuring that affordability constraints are taken into account) (p28)

increased private sector participation in commercial transport services (p28)

introduction of sustainable transport policies and best practice (p32)


Assumptions Inputs / Assets Activities Outputs: Outcomes: Impacts

SOEs as well as development of the framework for PPPs

Policy dialogue effectively coordinated with other

IFIs

Donors are willing to

support EBRD’s TC and

policy dialogue needs &

objectives

EBRD Policies

env & soc policy

policy for the financing of private parties to concessions

information policy

Focus 2: Sustainability

Environmental improvements implemented (3.2.2)

low carbon modes of transport developed (p31-32)

avoid: logistics, intermodal transport, incl. postal services (repeat of output 2)

shift: e.g. rail or inland waterways

improve: application of EE technologies, operational practices and standards

the Transport SEI/GET contribution doubled to 25% of Transport ABI (SPI 3)

EBRD E&S policy and EU standards adhered to in the areas of env appraisal, pollution prevention and abatement, biodiversity, resettlement and livelihood restoration, stakeholder engagement.

Social improvements implemented (3.2.3)

road safety infrastructure improvements undertaken economic inclusion and gender components implemented

Improved transport sustainability, safety and security (section 3)

energy consumption in transport services reduced

CO2 emissions in transport sector reduced (SPI)

safety in the transport sector improved (through improved road safety throughout EBRD’s ops) (p22)

greater inclusion in the transport sector (reduced disparities in access to employment, particularly for women and vulnerable groups or those living in remote rural or underserved urban areas) (p34)

- reduced number of road and rail accidents;

- improved access to employment, including local workforce participation in the construction phases of large transport projects, access to public and other services, skills transfer

Risks to Provision of Inputs: Risks to Achievement of Outputs: Risks to Achievement of Outcomes:

EBRD does not find sufficient opportunities to

invest

Lack of donor funding for TCs

Lack of sufficient EBRD staff resources

Lack of cooperation between IFIs

Political constraints limiting sector reforms

Potential clients’ unwillingness to work with EBRD

IFIs and EU have limited capacity or appetite for

co-financing

Weak institutional capacity or unwillingness of

public institutions to engage with EBRD

Delays, operational failures during project

implementation

Capital market ‘freeze’ restricts available co-financing

Resistance to change (public institutions; utilities)


Annex 3. Portfolio Analysis

Though the TSS was formally approved in October 2013, the first draft of the new strategy was prepared very early in 2013

(section 2). Discussions with the Management during the preparation of the approach paper have confirmed that the draft

strategy was used to guide the formulation of all projects prepared and approved from the start of 2013 onwards.

Moreover, Management considers TSS as an evolution of the approach the Bank had previously been taking in the sector,

and that the formal final approval of the strategy did not represent “a sudden change of course”. Commencing from the start

of 2013 would also:

widen the evaluation portfolio to include more mature projects considering the longer implementation timelines for

projects in the Transport sector.

continue from where the previous transport sector evaluation left off - end of 2012 (Annex 4).

An additional challenge for the evaluation to manage is that the TSS is still active, so as projects continue to be approved

and signed, they are done so under the TSS but too late be considered under this evaluation. Therefore cut-off point from

which the evaluation begins is required. The evaluation proposes that cut-off point to be the end of May 2017.

The population taken into consideration for the purpose of this evaluation is the portfolio of the Transport operations

managed by two teams: Transport and Infrastructure, Russia and Central Asia.

The timeframe under consideration is all projects approved and signed between January 2013 when the TSS was first

drafted and signed as at end of May 2017. The above approach has been discussed and validated by Management –

though discussions continue on cases where the application of the criteria is not straightforward.

Between January 2013 and May 2017, the EBRD approved and signed a total of 70 transport operations.

a. Investment operations

Table 5: TSS operations and net cumulative investment by Banking team (Jan 2013 – May 2017)

Banking Team Number of ops % Investment (€) %

1. TRA 51 73% 2,519,034,653 79%

2. INF RCA 19 27% 668,790,763 21%

Total 70

3,187,825,416 Source: DataWarehouse

Table 6: TSS operations and investment by status (Jan 2013 – May 2017)

Status Number of ops % Investment (€) %

Active 64 91.43% 3,081,511,306 96.66%

Complete 6 8.57% 106,314,110 3.34%


Table 7: Share of TSS operations and investment by stage and status (Jan 2013 – May 2017)

Status Life cycle stage % of ops % of investment

Active Disbursing 21.43% 36.66%

Repaying 41.43% 34.07%

Signed 28.57% 25.93%

Complete Completed 8.57% 3.34%



.

Table 8: TSS operations and investment by portfolio class (Jan 2013 – May 2017)

Portfolio Class Number of ops % Investment (€) %

Private 41 58.57% 1,264,026,212 39.65%

State 29 41.43% 1,923,799,204 60.35%


Table 9: TSS operations and investment by sovereign risk (Jan 2013 – May 2017)

Number of ops % Investment (€) % Average project size

Non-Sovereign 48 69% 1,528,283,695 48% 31,839,244

Sovereign 22 31% 1,659,541,721 52% 75,433,715


Table 10: TSS operations and investment by instrument (Jan 2013 – May 2017)

Instrument Type Number of ops % Investment (€) %

Debt 63 90% 3,021,800,632 95%

Debt & Equity 2 3% 30,060,071 1%

Equity 5 7% 135,964,713 4%


Table 11: TSS operations and investment by region and country (Jan 2013 – May 2017)

Region Country Number of ops % Investment (€) %

Central Asia (CA) Kazakhstan 14 20% 520,837,342 16%

Kyrgyz Rep. 1 1% 4,189,508 0%

Turkmenistan 1 1% 935,954 0%

Total 16 23% 525,962,804 16%

Central Europe and Baltics (CEB)

Croatia 3 4% 249,600,000 8%

Hungary 2 3% 30,060,071 1%

Poland 4 6% 77,707,894 2%

Slovak Rep. 1 1% 148,427,539 5%

Total 10 14% 505,795,504 16%

Cyprus and Greece (CYP & GRE)

Cyprus 1 1% 10,138,610 0%

Greece 2 3% 186,733,207 6%

Total 3 4% 196,871,817 6%

Eastern Europe and Caucuses (EEC)

Armenia 1 1% 27,914,204 1%

Azerbaijan 1 1% 18,184,249 1%

Belarus 2 3% 30,000,000 1%

Georgia 2 3% 6,750,000 0%

Moldova 2 3% 175,000,000 5%

Ukraine 4 6% 92,614,877 3%

Total 12 17% 350,463,331 11%

Russia (RUS) Russia 3 4% 142,827,959 4%


Total 3 4% 142,827,959 4%

Southern and Eastern Europe (SEE)

Albania 1 1% 36,870,000 1%

BiH 4 6% 231,000,000 7%

Bulgaria 1 1% 5,000,000 0%

FYR Macedonia 4 6% 315,150,000 10%

Kosovo 2 3% 48,200,000 2%

Montenegro 2 3% 25,000,000 1%

Serbia 1 1% 100,000,000 3%

Total 15 21% 761,220,000 24%

South-eastern Mediterranean (SEMED)

Egypt 2 3% 161,655,391 5%

Morocco 1 1% 200,000,000 6%

Total 3 4% 361,655,391 11%

Turkey (TUR) Turkey 8 11% 343,028,610 11%

Total 8 11% 343,028,610 11%


Table 12: TSS operations and investment by subsector (Jan 2013 – May 2017)

Subsector Number of ops % Investment (€) %

Road Construction 17 24% 1,151,050,573 36%

Intermodal 16 23% 429,794,201 13%

Rail Transport 14 20% 579,116,150 18%

Ports & Harbours 7 10% 329,539,437 10%

Air Transport 4 6% 225,797,411 7%

Support Activities 4 6% 155,628,890 5%

Rail Rolling Stock 3 4% 194,500,000 6%

Postal Service 2 3% 13,059,113 0%

Water Transport 2 3% 91,155,391 3%

Ship & Boat Building 1 1% 18,184,249 1%


Table 13: TSS operations and investment by environmental category (Jan 2013 – May 2017)

E&S Env Category Number of ops % Investment (€) %

A 7 10% 784,874,084 25%

B 59 84% 2,133,239,561 67%

C 1 1% 89,138,477 3%

NA 3 4% 180,573,294 6%


Table 14: TSS operations and investment by TI rating at approval (Jan 2013 – May 2017)

TI rating at approval Number of ops % Investment (€) %

Strong Good 10 14% 536,274,538 17%

Good 38 54% 2,050,453,391 64%


Moderate Good 7 10% 344,297,098 11%

Satisfactory 1 1% 11,150,000 0%

N/A 14 20% 245,650,390 8%


Table 15: TSS operations and investment by SEI Flag (Jan 2013 – May 2017)

SEI/GET Flag Number of ops Investment (€) SEI/ GET Finance

N 35 1,437,469,880 0

Y 35 1,750,355,536 1,030,353,219

Total 70 3,187,825,416 1,030,353,219


b. Technical Cooperation (TC)

Table 16: Transport TCs by year (Jan 2013 – Dec 2016)

Year Number of TCs Amount (€)

2013 31 6,669,677

2014 26 11,060,000

2015 23 4,261,000

2016 31 12,360,050

Total 111 34,350,727

Source: Data from Infrastructure TC Unit

Table 17: Transport TCs by country (Jan 2013 – Dec 2016)

Country Number of TCs Amount (€)

FYR Macedonia 16 6,605,000

BiH 10 1,984,000

Kazakhstan 10 2,280,000

Morocco 8 2,438,190

Moldova 7 2,008,000

Montenegro 7 1,415,000

Croatia 7 1,262,000

Albania 7 2,862,000

Kosovo 7 5,540,000

Russia 6 760,677

Egypt 6 2,345,000

Total 91 29,499,867


Table 18: Transport TCs by transaction relation (Jan 2013 – Dec 2016)

TC Type Number of TCs % Amount (€) %

Transaction related 102 92% 31,230,050 91%

Non-transaction related 9 8% 3,120,677 9


Total 111 34,350,727


Table 19: Transport TCs by donor (Jan 2013 – Dec 2016)

Donor Number of TCs Amount (€)

EBRD Shareholder Special Fund 51 11,868,860

Western Balkans Investment Framework 8 9,352,000

Not provided 13 2,290,000

EBRD Shareholder Special Fund / Early Transition Countries Fund

3 2,073,000

Central European Initiative 7 1,748,000

To Be Confirmed 8 1,509,000

Global Environment Facility (GEF) 4 1,221,867

Kazakh TC Fund 1 1,200,000

Japanese Technical Cooperation Fund 1 885,000

SEMED Multi Donor Account 1 500,000

Swedish Technical Cooperation Fund 5 495,000

French Technical Cooperation Fund 4 400,000

Czech Technical Cooperation Fund 1 350,000

SEMED Cooperation Fund 1 250,000

SEMED Project Preparation Facility 1 71,000

SSF 1 70,000

Taiwan Technical Cooperation Fund 1 67,000

Total 111 34,350,727


Table 20: Transport TCs by subsector (Jan 2013 – Dec 2016)

Subsector Number of TCs Amount (€)

Rail 56 21,135,000

Road 27 5,366,000

Ports 11 3,108,190

Advisory/Policy 4 2,700,000

Post 3 874,000

Aviation 6 801,860

Shipping 3 293,677

Intermodal 1 72,000

Total 111 34,350,727



Annex 4. Relevant evaluation work

The evaluation team identified a number of related evaluations by EvD in recent years that provide input, or upon which this

evaluation can build.

EvD Special Study on the Transport Operations Policy, 2011

As mentioned earlier, EvD produced an evaluation of the Bank’s TOPs – the first such exercise for the Bank’s transport

sector – with a coverage that included all Bank policies from 1992, 1997 and 2005 (0).

Based on a comparison of the evaluation results in the transport sector with the Bank’s average rating scores as provided in

the latest Annual Evaluation Report (AER) at the time, the study found the following:

Additionality, fulfilment of objectives, and company financial performance match the Bank’s average

performance ratings,

Project financial performance and environmental performance/change perform better than the Bank’s average,

and

Transition impact, and Bank handling perform below the Bank’s average

Moreover, the study made five recommendations that the Bank’s next Transport Sector Policy include:

The Bank needs to more realistically align overall sector reform expectations with individual project

expectations and deliverables.

The Bank needs to place more emphasis on MDB/Donor cooperation.

In light of climate change implications, the new TOP should place more emphasis on environment and

commensurate technologies.

A more holistic or indeed ‘integrated’ approach needs to be adopted by the Bank in infrastructure (transport)

projects during project preparation.

Sector policies accountability needs strengthening.

Management considered generally that the recommendations are reasonable and in most cases are already being

implemented. Audit Committee minutes show that

underlined the Committee’s request for better recognition of the role of TC in policy dialogue and the need to

clearly define what the Bank wanted to achieve through policy dialogue.

the Committee had also stressed the need for close cooperation between MDBs in the sect

APPROACH PAPER Transport Sector Strategy: EvD Review · 2021. 1. 27. · EvD Approach Paper 4 1....

Documents

Transcript of APPROACH PAPER Transport Sector Strategy: EvD Review · 2021. 1. 27. · EvD Approach Paper 4 1....