Joint Evaluation of the International Trade Centre documents/Different... · 2017-05-09 · Joint...

Joint Evaluation of the International Trade Centre

Version Final pg 1/142

Joint Evaluation of the

International Trade Centre

File No. 104.A.1.e37

Draft Final Inception Report

August 20, 2004

DMI ASSOCIATES

in association with

ACE-Global, Ticon/DCA, Copenhagen DC

21, rue Longue

Lyon, France



Table of contents

1 INTRODUCTION..............................................................................................................................6

1.1 PROJECT SYNOPSIS: JOINT EVALUATION OF THE INTERNATIONAL TRADE CENTRE..................6

1.2 OUTLINE.......................................................................................................................................6

2 BACKROUND AND RATIONALE ...............................................................................................7

2.1 OVERVIEW OF THE ITC ................................................................................................................7

2.1.1 Mission Statement...................................................................................................................7

2.1.2 Objectives of ITC.....................................................................................................................7

2.1.3 Origins and governance ..........................................................................................................8

2.1.4 The last Decade........................................................................................................................8

2.1.5 Strategic Issues......................................................................................................................13

2.2 OBJECTIVES OF THE EVALUATION .............................................................................................17

2.3 KEY EVALUATION QUESTIONS FROM THE TORS ......................................................................18

3 METHODOLOGY ...........................................................................................................................19

3.1 INTRODUCTION ..........................................................................................................................19

3.2 DESCRIPTION AND CHOICE OF METHODOLOGIES ....................................................................19

3.2.1 Uniqueness of ITC.................................................................................................................19

3.2.2 Stakeholders...........................................................................................................................20

3.2.3 Approach to the evaluation....................................................................................................22

3.3 OVERVIEW OF THE METHODOLOGY ..........................................................................................24

3.4 THE EVALUATION MATRIX.......................................................................................................26

3.4.1 TRTA Results-Based - reference framework.........................................................................26

3.4.2 ITC - Results-Based logic of intervention .............................................................................32

3.4.3 The key indicators..................................................................................................................33

3.4.4 The key evaluation questions.................................................................................................37

3.5 IMPLEMENTATION MODALITIES................................................................................................38

3.5.1 Assessment of ITC relative position in TRTA and of Developed Country Donor/TSI

Perspectives..........................................................................................................................................39

3.5.2 Assessment of ITC performance and comparative advantage at the Country or Regional

Level 40

3.5.3 Assessment of ITC Products..................................................................................................40

3.5.4 Assessment of Corporate Organisation, Financing and Management..................................42

3.6 DATA GATHERING TOOLS ..........................................................................................................43

3.6.1 Sampling................................................................................................................................46

3.6.2 Evaluation steps ....................................................................................................................47

3.7 APPROACH TO THE EVALUATION .............................................................................................48

3.8 COMMUNICATIONS STRATEGIES ...............................................................................................49

3.8.1 Communication with Stakeholders........................................................................................49

3.8.2 Public Communications and Website....................................................................................49

3.9 QUALITY ASSURANCE ...............................................................................................................56

3.10 RISKS, MITIGATING STRATEGIES AND ASSUMPTIONS...............................................................56

3.10.1 Risks:.................................................................................................................................56

3.10.2 Mitigation strategies.........................................................................................................57



3.10.3 Assumptions .....................................................................................................................58

3.11 ISSUES/CONCLUSIONS................................................................................................................58

4 PROJECT PLANNING....................................................................................................................59

4.1 WORKPLAN ................................................................................................................................59

4.1.1 Methodology, Timeframe and outputs...................................................................................59

4.1.2 Tasks, Milestones and Reporting ..........................................................................................60

5 RECOMMENDATIONS.................................................................................................................62

5.1 TIME FRAME...............................................................................................................................62

5.2 BUDGET ......................................................................................................................................62

5.3 COMMUNICATIONS....................................................................................................................62

5.4 MOBILISATION OF PERSONNEL..................................................................................................62

6 ANNEXES..........................................................................................................................................64

6.1 ANNEX 1: EVALUATION QUESTIONS.........................................................................................64

6.2 ANNEX 2: INDICATORS MATRIX.........................................................................................82

6.2.1 Project Universe for Evaluation ..........................................................................................104

6.3 NOTE ON COUNTRY SELECTION..............................................................................................113

6.4 ANNEX 4: QA PLAN ................................................................................................................122

6.4.1 Introduction to the Quality Assurance Management System ............................................122

6.4.2 Administrative procedures ..................................................................................................122

6.4.3 QA Annexes ........................................................................................................................129

6.4.4 QA Annex 1 ........................................................................................................................131

6.4.5 QA Annex 2 QA of Reports ................................................................................................133

6.4.6 QA Annex 3 Template for Reports......................................................................................136

6.4.7 QA Annex 4 File Structure.................................................................................................137

6.4.8 Structure of the WEB Server...............................................................................................140



Document control

Name Function Date Signature

Author MGS, AF, AL,

NJG, ME,

SVD

TL,

Evaluators,

Approved by

(if relevant)

ACRONYMS AND ABBREVIATIONS

ACP African, Caribbean & Pacific States

APEC Asia-Pacific Economic Co-operation

CCA Common Country Assessment

CIS Commonwealth of Independent States

CO Country Office

CSO Civil Society Organization

EOI Expression of Interest

EU European Union

FDI Foreign Direct Investment

FTA Free Trade Area or Free Trade Agreement

GATS General Agreement on Trade in Services

GATT General Agreement on Tariffs & Trade

GSP Generalised System of Preferences

IF Integrated Framework

IFIS International Financial Institutions

ITC International Trade Centre

JITAP Joint Integrated Technical Assistance Program

LDC Least Developed Countries according to UN designation

NAFTA North American Free Trade Area

NHDR National Human Development Report

NPO National Programme Officer

NGO Non-Governmental Organization

OECD Organization for Economic Co-operation and Development

PRSP Poverty Reduction Strategy Paper

RBM Results-Based Management

ROAR Results-Oriented Annual Report

SHD Sustainable Human Development

SME Small or Medium-sized Enterprise



ACRONYMS AND ABBREVIATIONS

SPS Sanitary and Phyto-Sanitary

SRF Strategic Results Framework

SSE Small-Scale Enterprises

TBT Technical Barriers to Trade

TRA Trade Related Assistance

TRIPS Trade Related Intellectual Property Rights

TORS Terms of Reference

TRTA/CB Trade Related Technical Assistance & Capacity Building

UN United Nations

UN/CEFACT United Nations Centre for Trade Facilitation & Electronic Business

UNCTAD United Nations Conference on Trade & Development

UNDAF United Nations Development Assistance Framework

UNECE United Nations Economic Commission for Europe

UNDP United Nations Development Programme

UNIDO United Nations Industrial Development Organisation

UNISTE United Nations International Symposium on Trade Efficiency

WCO World Customs Organization

WTO World Trade Organization

GLOSSARY1

Indicator: Signal that reveals progress (or lack thereof) towards objectives; means of

measuring what actually happens against what has been planned in terms of

quantity, quality and timeliness. It is a quantitative or qualitative variable

that provides a simple and reliable basis for assessing achievement, change

or performance.

1 Note that we are using the OECD Glossary of Key Terms in Evaluation and Results Based Management,

2002.



1 INTRODUCTION

1.1 Project Synopsis: Joint Evaluation of the

International Trade Centre

Project number: File No. 104.A.1.e37

Country : Global

This project is the Joint Evaluation of the International Trade Centre (ITC). It is

supported by a group of seven donors led by Danida. The purpose is to assess the

comparative advantages of the ITC, to evaluate the performance of the ITC’s

interventions, and to assess the capacity of the ITC and Donor-Funding Modalities. The

project is organized in four phases. The first phase, Inception and Desk Study, is from 5

January until 31 March 2004. The second phase, Field Studies and Technical Studies, is

from 15 April 2004 until September 30. The third phase, which overlaps with the second

phase, is the preparation of the validation of the technical reports and is from 15

September through November. The fourth and final phase, the presentation of the initial

draft synthesis report and the preparation of the final report is to be completed by

January 2005.

The project involves a team of international evaluators and a group of local and regional

evaluators will be added to the team. The evaluation is an external activity-based

evaluation, which examines the international context and the experience and

performance of ITC over the last decade and which will make recommendations for the

future.

1.2 Outline

The next section of the Draft Final Inception Report, Chapter 2, examines the

Background and Rationale for the Evaluation. The background analysis in the Draft

Final Inception Report is supplemented by, and draws upon, the Desk Study, which

examines the context for, and the key activities of, the ITC in more depth.

Chapter 3 presents the evaluation methodology. Chapter 3 together with Annexes 1, 2,

and 3, presents the evaluation matrix.

Chapter 4 summarises the planning for the evaluation and Chapter 5 summarises

recommendations.



2 BACKROUND AND RATIONALE

2.1 Overview of the ITC

This section provides a brief overview of the Mission, Objectives and Activities of the

International Trade Centre (ITC). The ITC is jointly sponsored by the UNCTAD and the

WTO.

2.1.1 Mission Statement

“ITC is a technical cooperation organization whose mission is to

support developing and transition economies, and particularly their

business sectors, in their efforts to realize their full potential for

developing exports and improving imports operations with the

ultimate goal of achieving sustainable development. ITC deals

specifically with the operational aspects of trade promotion and

export development.”

ITC mission statement / Mandate, www.intracen.org.

2.1.2 Objectives of ITC

The latest ITC’S strategic and specific objectives have been gathered from 2 sources: The

Business Plan 2003-2005, and the Budget program 2004-2005. The strategic level of

objective defines the broader context in which ITC, as an organisation, operates. This

broader context is mostly framed through the Millennium Development Goals (UN) and

the Doha Development Agenda (WTO).

The corporate level of objective for ITC defines the added value of ITC, the specific goals

it focuses on. At the present time the ITC has specified five long-term corporate goals,

i.e.:

Goal 1: To facilitate the integration of developing and transition economy

enterprises into the MTS;

Goal 2: To support national efforts to design and implement trade development

strategies;

Goal 3: To strengthen key trade support services, both public and private;

Goal 4: To improve sector export performance in sectors of critical importance

and opportunity;

Goal 5: To foster international competitiveness within the business community

as a whole and the SME sector in particular.



2.1.3 Origins and governance

The International Trade Centre (ITC) was created in 1964 as an affiliate to the GATT and

since 1968 has been operated jointly by GATT (replaced by the WTO in January 1995)

and the United Nations through UNCTAD. ITC is responsible in the UN system for

implementing technical assistance projects in developing countries and economies in

transition related to trade promotion with a view to expanding their exports and

improving their import operations. ITC mandate gives priority to supporting trade

development efforts of the least developed countries with, more recently, a focus on

transition economies. ITC receives its core funding from the UN and WTO, and is also

funded by donors and by UNDP through ITC’s trust funds.

2.1.4 The last Decade

Following a protracted period in the early 1990s, during which the ITC suffered from a

lack of direction, and which led to a significant drop in funding, a major priority–setting

exercise was undertaken in 1994. The outcome of this was the publication of the initial

strategic orientation document entitled ITC at the Crossroads, an Agenda for the Future,

which was presented to the Joint Advisory Group (JAG) in November 1994.

Subsequently, the new priorities were approved by the JAG in May 1995 in the Priority

Setting Review.

1994-95 Strategic Review

The 1994-95 review acknowledged the fact that developing countries generally were

being left behind in terms of their share of, and participation in, world trade. The

economic reform programmes implemented at the time were not in and of themselves

sufficient – they needed to be based on effective trade development strategies and

reinforced with trade support initiatives. At the same time, the approach to technical

assistance had changed substantially since the formation of ITC 30 years previously, and

many new providers of TA had emerged.

ITC’s technical cooperation programme was identified as comprising six core

services or areas of intervention, specifically:

• Trade development strategy and design;

• Trade Information;

• Export product and market development;

• Development of trade support services;

• Import management;

• Human resource development.



Within each of these six core areas of intervention, various priority activities were then

identified.

The main elements for revitalising ITC were summarised as follows:

• Becoming more demand driven and enhancing ITC’s orientation towards

recipients;

• Strengthening ITC’s core services;

• Fully mobilising ITC’s human resources (e.g. through interdivisional

collaboration);

• Restructuring and streamlining ITC’s operations, by modifying the

organisational structure and related measures;

• Enhanced mobilisation of financial resources through measures such as the

proposal for a Global Trust Fund, a review of donors priorities, and other

initiatives to reduce the dependence on UNDP.

As part of the restructuring, and in order to increase the resources available for technical

cooperation with developing countries and countries in transition, the Global Trust

Fund was established in 1995. This is the primary channel through which donor country

funding is channelled, and it forms the primary source/pool of financing for ITC field

operations.

Clearly the immediate impetus to the 1994 Strategic review was the internal

management challenges and the sharp decline in funding reflecting both issues specific

to ITC and broader issues in the UN system. However, the external context for the ITC

was changing in the 1990s. It is worth noting that the Priority Setting Review took note of

the external context and explicitly raised the issue of the comparative advantage of the

ITC.

The Priority Setting Review adopted by the JAG stated:

“ITC will devote resources only to areas in which it has

comparative advantage. Among those identified in the Agenda (for

the Future) paper are: its substantive expertise in trade promotion

and export development for developing countries; its impartiality

and multilateral approach; its access to policy- and decision-makers

in developed and developing countries; its thorough

understanding, born of long and extensive experience, of the

interlinkages between the global, macro- and microeconomic

aspects of international trade; its proven ability to formulate and

implement trade promotion and export development programmes;



and its wide-ranging contacts. Further prioritization of ITC’s future

areas of specialization was made on the basis of the following

criteria: relevance to the most pressing needs of recipient countries;

the likelihood of making a visible contribution to trade

development efforts; the probability of attaining a critical mass of

related technical expertise in ITC; the possibility of achieving an

acceptable expenditure/impact ratio (cost-effectiveness); and the

potential for generating extrabudgetary resources for activities at

the country level.”

Priority Setting Review, pp. 1-2, paragraph 4

If we are to consider what the comparative advantage of ITC is today and will be in the

future, we need to consider how the external context for the ITC has evolved and is

likely to evolve in the future.

The External Context in the 1990s

In the mid-1990s the role of the ITC evolved due to external factors including

developments in the global trading system and broader economic and political

developments. Key developments include the following:

• the fall of the Berlin Wall and the collapse of the Soviet Union created a new set of

challenges to support the integration of transition economies into the global system;

• new regional trading initiatives, including the North American Free Trade

Agreement and the enlargement of the European Union, created new competitive

pressures for trade and investment;

• new concerns about environmental, technical and sanitary and phytosanitary

standards in the markets of developed countries;

• the development and widespread application of new information and

communications technologies; and

• the completion of the Uruguay Round of multilateral negotiations and the creation of

the World Trade Organization (WTO) on January 1, 1995, with a more comprehensive

set of multilateral trading rules.

Other important developments during the 1990s, but which may have received less

attention at the time, included:

• the large queue of about 30 countries including China wishing to accede to the

WTO;



• the implications, for governance and for the business communities in developing

and transition economies, of the developments of the Multilateral Trading System

(MTS);

• the links between trade enabling policy environment and trade development; and

• the broader social and development impacts of trade development

Within this overall context, the developments in the MTS were of greatest direct

relevance to the evolution of the role of the ITC. After the creation of the WTO in 1995,

the role of the ITC was adapted to the new configuration of international organisations.

In particular, there were discussions among the WTO, UNCTAD and the ITC about their

respective roles, and new approaches were developed for co-operation among these

agencies.

The Joint Integrated Technical Assistance Program, (JITAP), which was a specific

response to a proposal at the meeting of African Trade Ministers in Tunis in October

1994, reflected a different approach to supporting African LDCs to meet the challenges

of participation in the trading system with the creation of the WTO. The JITAP, which

involved collaborative identification activities among the organisations, was launched as

a joint activity of the WTO, UNCTAD and the ITC in 1998. The WTO and UNCTAD are

partners with ITC in JITAP, which in phase I operated in eight African countries, and an

additional eight countries were added in phase II launched in 2003. A broadly positive

evaluation of JITAP was carried out in 2002, although a number of issues were raised in

the evaluation.

The launch of the Integrated Framework (IF) represented a similar shift of approach, in

recognition of the shortcomings of a piecemeal approach to trade related development.

The political impetus for the IF came from the 1996 Singapore WTO Ministerial

Declaration, which called for a high level meeting on LDCs involving the WTO,

UNCTAD and the ITC. The IF was formally launched at the high-level meeting on

LDCs at the WTO in late 1997. The IF recognises the interdependencies involved and the

need for an integrated approach, with the ITC’s emphasis on trade promotion combined

with the respective strengths of the World Bank, the IMF, WTO, UNCTAD and UNDP.

In turn the IF is also intended to be a catalyst for bilateral TRTA projects in the

participating countries.

Recent Developments and the Present Context

The context within which the ITC operates today is different from that of 10 years ago in

some important respects. Some trends have intensified while other developments have

declined in significance. The overall strategic context is considered more extensively in

the desk study and this section only summarises the key points.



The most directly relevant development to the technical assistance activities of ITC is the

debate about the links between trade and development and the expansion of TRTA.

There is now within the mainstream international development community a much

greater acceptance of the importance of trade enhancement and of business sector

support as development tools. At the same time, there has been a substantial expansion

in trade related technical assistance (TRTA), which now runs to several billions of

dollars annually (and of which the ITC expenditure constitutes a very minor share).

In terms of the specific funding of ITC, these changes are manifest in shifts in the sources

of funding. In contrast to ten years ago (or fifteen years ago when the budget was

larger), the ITC operations are now much more dependent upon bilateral donors and the

funding from UNDP has declined significantly.

A notable recent development has been an increase in the variety and number of donor

countries coming forward, following many years in which the Scandinavian countries

and the Netherlands comprised the core-funding base. For example, though on a small

scale, the USA funding commitment in 2003 could be seen as a breakthrough. Also, after

many years of expectancy, the EU has committed €5 million over 3-4 years under an

Asia Trust Fund. Switzerland, the UK, Canada, France, and Germany are now much

more to the fore than in earlier years.

According to the most recent documentation available, the regional expenditure

breakdown of operations funds is as follows overleaf:



Regional Distribution of Activities 2001 2002

Africa - 43% Africa - 43%

Asia/Pacific - 13%

Asia/Pacific - 15%

Latin America and the Caribbean -

13%


13%

CEE and CIS - 19%

CEE and CIS - 14%

Arab States - 5% Arab States - 7%

Global - 7% Global - 8%

Types of services provided

According to the ITC Business Plan 2003-2005, the ITC currently delivers its services

under seven programme headings, as follows:

• Strategic and Operational Market Research: It is designed to support priority

setting by trade strategists and managers of trade support institutions.

• Business Advisory Services: This is aimed at supporting participation by

developing countries in the MTS, with the support to participation in the MTS

being coordinated under the World Trade Net system, and at supporting

competitiveness at the enterprise level.

• Trade Information Management: This programme concentrates on capacity

building and networking activities within LDCs in the trade information area.

• Export Training Capacity Development: This programme focuses on

developing the competencies of enterprise managers to ‘nurture’ their export

business.

• Sector Specific Product and Market Development: This programme focuses on

supply-side constraints (e.g. high costs of doing business) to improving a

sector’s export performance.

• Trade In Services: The focus here is increasing awareness for service sector

opportunities and enhancing capabilities of services industries to participate in

international commerce.

• International Purchasing and Supply Chain Management: This programme

focuses on improving purchase and supply operations with the aim of

enhancing the reliability and competitiveness of export delivery.

2.1.5 Strategic Issues



The ITC faces a continuous challenge to appropriately position itself within a dynamic

trade policy and trade promotion environment. (These challenges are addressed in more

detail in the desk study and only the key points are summarised here.) Its programmes

and activities need to be continuously monitored to ensure ongoing relevance in this

changing environment. This evaluation will incorporate into its approach an assessment

of the key strategic considerations that the future evolution of ITC should

‘acknowledge’. Foremost among these are the following:

1) Developments in the Multilateral Trading System

The changing role of the ITC is linked to broader changes in the Multilateral Trading

System, which include the following developments:

• The mid-1990s saw the rapid evolution of the new multilateral trading system (MTS)

including the creation of the World Trade Organization (WTO) in 1995. The trading

rules were changed and expanded substantially under the WTO and developing

countries were subject to a fuller range of obligations under the trade rules, but

many of these obligations only came into effect after 2000 or even later, due to

transition provisions negotiated in the Uruguay Round. The accession of many

countries to the WTO has brought many countries fully into the MTS for the first

time.

• Related to the wider application of trading rules under the WTO there was

considerable apprehension in developing countries and more particularly in the

least-developed and low-income countries and in civil society in developed and

developing countries, regarding the implications of the World Trade Organisation

and the rules which would be negotiated and applied. These concerns still dominate

the landscape of International trade and are important factors influencing the

negotiations on the Doha Development Agenda.

• These concerns and negative perceptions of some developing countries have

manifested themselves in different ways at the Seattle, Doha and Cancun Ministerial

meetings of the WTO. At the Seattle Ministerial Meeting the concerns of developing

countries and anti-globalisation activists resulted in a blockage of the negotiations. In

contrast, at the Doha Ministerial Conference the result was a complex negotiating

agenda, -- the Doha Development Agenda (DDA) which attempted to respond to the

concerns of developing countries about WTO implementation issues associated with

the implementation of the Uruguay Round commitments and to negotiate new

trading rules for developing countries. The inconclusive outcome of the Cancun

Ministerial Meeting reflects the complex challenges of integrating developing

countries into the trading system and differences of view on how to adapt the

trading rules resulting from the Uruguay Round and on how to deal with difficult

challenges such as negotiations on agricultural trade.

The ITC’s mission includes assisting with the integration of developing countries into

the MTS, and supporting the business communities to understand the significance of,



and participate in, the development of trade policies and to prepare for international

negotiations. This has been reflected in new activities of the ITC such as regional

business fora on trade negotiating issues.

2) Developments in regional integration

Delays in the multilateral negotiations under Doha, do not block, and may also act to

stimulate, regional trading initiatives with important implications for developing

countries. Examples of such initiatives include:

• The process of enlargement of the European Union to Central and Eastern

Europe;

• Implementation of existing regional arrangements such as NAFTA and AFTA.

• The prospective negotiation of regional trading arrangements between China

and AFTA;

• Negotiation of bilateral FTAs by the EU with trading partners such as Mexico as

well as negotiations with MEDA partners;

• The negotiation of bilateral FTAs by the United States with various countries or

groups of countries including Central American countries, Singapore and

Thailand respectively;

• The prospective negotiation of the Economic Partnership Agreements (EPAs)

under the Cotonou Agreement between the EU and the ACP countries;

• Prospective regional arrangements in APEC and the Americas;

• Various regional initiatives in Africa; and

• Negotiations within SAARC in South Asia.

3) The recognition of linkages in the role of trade development and the broader process

of economic development.

• The expansion of TRTA globally and by multiple agencies reflects the recognition

of the link between trade and broader sustainable development objectives.

• A re-examination of the link between trade and human development is

underway, both within and outside the UN. The widening human development

mandate in terms of poverty alleviation has potential implications for the process

of trade development.

• While the process of globalisation has given added urgency to the need to

address issues of trade and poverty in LDC countries, the MTS has also

strengthened the global consensus, in the developed world and among the

donors and the development assistance community, that trade, poverty,

development and capacity-building issues need to be addressed together, in a

holistic manner, thus promoting the integrated approach. The Integrated



Framework, for instance, is a novel and innovative approach to development

assistance that reflects this new approach2.

• Related to changing approaches to development are the changing approaches of

donors to support the multilateral technical assistance organisations and

international financial institutions and the changing approaches of donors to the

programming bilateral technical assistance activities.

4) The widespread diffusion of information and communications technologies

The development and application of information technologies has important

implications for developing countries and direct relevance to the core competencies of

ITC. Among relevant factors are:

• The disparities in telecommunications infrastructure and Internet connectivity

between developed and developing countries and the associated technology

deficit and digital divide between the developed and developing world, and the

impact that has on ITC’s mission.

• The impact of e-commerce in reshaping global trade patterns and possibilities

(e.g. in reversing long standing situational disadvantages, c.f. outsourcing of IT

services).

• The changes in the relative costs of, and access to, business and trade

information.

5) Broader changes in business strategies and practices in response to changes in the

trade and business environment and in response to the competitive opportunities

and challenges associated with the new technologies.

Some of the most prominent factors are:

• The growing interconnectedness of investment and trade flows. Foreign Direct

Investment can be a powerful tool for trade development and FDI has been very

important in supporting the expansion of exports in regions as diverse as China,

Mexico and the Central European economies acceding to the European Union.

Over the last decade Foreign Direct investment has grown much more rapidly

than international trade, which in turn has grown more rapidly than national

economies.

• The development of global supply chains, which have important implications for

both marketing and business organisation and quality standards.

2 The integrated Framework aims at linking trade to development and poverty reduction by coordinated

interventions of all the major actors of development assistance, namely: the World Bank, the IMF, the

WTO, the UNDP, UNCTAD and ITC. One of the innovative aspects of the project is the participatory

approach, where beneficiaries are fully engaged in the development process by linking technical assistance

to their national strategies of poverty reduction, and where bi and multi lateral donors are mentors to the

process.



• The appropriateness of ITC’s focus - in particular on SMEs and TPOs in light of

the changing technologies and business practices.

All of these factors are relevant to the consideration of the present and future role of the

ITC.

If, for example we focus on the last point mentioned above, the role of national TPOs is

constantly evolving in the new, e-commerce and internet-based, trading environment.

National TPOs have been the main organisational counterpart to the ITC in beneficiary

countries. ITC’s relationship with these organisations must be examined, and it must be

determined whether they remain the optimal vehicle for the achievement of ITC goals.

For example, changes in information and communications technology greatly facilitate

self-generated business trade ventures and research, independently of the support

network of TPOs.

Furthermore, given the increasing interconnectedness of investment and trade, should

national investment agencies be brought more to the fore? In many countries trade

promotion and investment promotion functions have been merged, yet ITC does not

address investment promotion. In many countries also, the client firm base of TPOs may

not be fully representative of the wider prospective export community.

These factors impinging on the strategic context for the ITC issues are explored in more

detail in the Desk Study, but our results clearly are preliminary at this stage. A scan of

the international context for the ITC is a major focus of this evaluation and will be a

major focus of the field and technical studies.

2.2 Objectives of the Evaluation

The objectives of the Evaluation are to learn from experience and to contribute to the

enhanced effectiveness of the ITC’s programs and activities.

The Terms of Reference state the following objectives:

• The first main objective of the evaluation is to determine ITC’s results,

and success in fulfilling its objectives (see … above) by measuring output,

outcome and impact to the extent possible.

• The second main objective is to determine the comparative advantage of

the organization within the international trade and development community,

which has changed significantly, especially during the past decade.



• The evaluation is further expected to offer recommendations on priority

setting and on steps ITC should take to improve its future performance.

2.3 Key Evaluation Questions from the ToRs

There are three broad sets of evaluative questions under the Terms of Reference

Q1: What is the Comparative Advantage of the ITC?

Q2: To assess the performance of ITC interventions and whether

they are achieving their objectives?

Q3: How effective is the management capacity of ITC and what are

the implications of donor-funding modalities?

Particular attention needs to be given to the latest developments of ITC interventions:

1. Old and new geographical coverage (issues of level from national to cross-border to

regional, evolution of transition economies, Doha Development Agenda)

2. Guiding principles and coordination with emerging common concerns (of the UN, of

the international donor community, Millennium Development Goals)

3. Impact of new information technologies on trade promotion activities

4. Old and new partnerships (exploring long term partnerships, new structures and

modalities). Evaluation of the comparative advantage of ITC in the post Doha

environment. Synergies and risks.

5. Enhancement of performance through state-of-the art management practices, use of

best practices as per lessons learned, use of evaluation as basis for enhanced

management.



3 Methodology

3.1 Introduction

We propose to use a mix of evaluation methods and tools. The primary evaluation

model will be the activity-based Results-Based Management Model; an Evaluation

Matrix is developed below to organize the evaluation of inputs, activities, outcomes and

impacts. Special attention will be paid to sampling in order to minimize biased

reporting. Where possible we will use triangulation to verify facts and observations by

obtaining different sources of information. Complementary tools and approaches

including the SWOT analysis and the Budgetary Analysis, will supplement the RBM

approach.

3.2 Description and Choice of Methodologies

3.2.1 Uniqueness of ITC

ITC appears to be unique in a number of respects, which we will strive to reflect in the

evaluation approach. These distinctive aspects are outlined below.

Development/business hybrid? ITC is a UN technical cooperation organisation whose

mission is to support developing and transition economies, and particularly their

business sectors, in their efforts to increase exports and to improve import operations.

As such, ITC combines a Development and Business mind set-, and terminology, tools,

evaluation criteria and management processes.

ITC in TRTA? ITC specialises in Trade Related Technical Assistance (TRTA), a field

undergoing rapid change since the commitment of Governments to the Doha

Development Agenda and the Millennium Development Goals (MDG).

No universally accepted and comprehensive common framework of what is TRTA

exists. Some indicators are provided with the recent development of the OECD database

on TRTA, which is one approach to benchmarking ITC interventions. This particular

framework however needs to be validated with the empirical realities of ITC

interventions and the limitations of the OECD data need to be taken into account.

With about 185 staff, and an annual core budget of around 33 million USD, ITC is a

relatively small UN organisation. However, it is not necessarily small when compared

to other TRTA partners and similar service providers. Yet it is working in many

countries and in different areas of competence.



A changing organisation! ITC has also been challenged in a number of different ways

and in different areas in recent years. Examples include the requirement to:

• adopt results-based management,

• respond to an increase in the number and diversity of requests from the field,

• rebuild credibility after the organisational crisis of the mid 1990s

• raise funds to increase its capacity,

• be recognized as a centre of excellence for trade development, with highest

standards of efficiency, competence, integrity and neutrality, and

• respond to broader issues in the evolution of the trading system and global

development concerns.

The organisation’s response to these pressures is reflected in on-going changes to

corporate documents, in the creation of new units, which reshape existing divisions, and

in the addition of new programmes to the seven primary ones. These developments

have had both positive and negative consequences: ITC appears to be very dynamic,

adaptive, and innovative, but this adaptability can also be confusing to newcomers and

donors, leading to such questions as: what has really changed, what is the real added

value of ITC, and where is ITC headed over the longer term?

These questions are very specific to ITC and have affected the methodology adopted for

the evaluation. We have adopted a TRTA result-based reference framework broken

down to the key common denominators of TRTA. This is presented as an open results-

based reference framework (not a blueprint), to permit elements captured during the

field study to enrich the reference framework of TRTA and better assess ITC’s position.

The overall strategy is to build upon the most recent conceptual models and tools of

development and business evaluations.

3.2.2 Stakeholders

ITC Stakeholders are those agencies and organisations that have a direct or indirect

interest in the ITC intervention or its evaluation3. Six types of stakeholders are identified:

• Parent body (WTO, UNCTAD)

• Donors to ITC, or to beneficiary countries (bilateral, regional development banks)

• Executive (Management and staff)

• Cooperation partners, peers, similar service providers

• Direct beneficiaries or clients of services provided by ITC (e.g. trade promotion

organisations or trade support institutions, business associations, or partner

3 OECD Glossary of evaluation terms



networks). Typically ITC works with intermediaries located at regional or

country levels to increase its outreach capacity.

• End-users (e.g. firms, enterprises, and particularly SMEs).

Stakeholders’ perspective is important to the evaluation of ITC as they all reflect

differently on the success or otherwise of ITC’s activities. Our approach is to integrate

the participation of stakeholders at each phase: design of the evaluation matrix, data

generation, analysis, and feedback.

Stakeholders and their Expectations.

Type of stakeholder Lists Expectations towards the

evaluation

Parent body - WTO,

- UNCTAD

- Comparative advantage of

ITC

Donors - Bilateral donors

- Regional

development banks

- Comparative advantage

- Performance

- Modalities of funding

- Transparency

Executive and staff - SMC,

- Individual ITC

Sections

- ITC’s position in TRTA,

Impact measurement,

funding stability

- Organisation structure

Partners, peers,

similar service

providers

- Multilateral agencies,

- Bilateral agencies,

- NGOs,

- Universities,

- private and public

sector in beneficiary

countries

To be identified during field

study

Direct beneficiaries - Public and private

Trade Support

Institutions (TSIs),

- business associations,

- chambers of

commerce,

- training firms,

- visible networks,

- trade and industry

associations,


study



- ITC liaison, etc.

(1500 registered in ITC’S

TSI database)

End-users Enterprises, particularly

SMEs


study

3.2.3 Approach to the evaluation

The approach to the evaluation will be (1) independent, (2) participatory, (3) output and

results oriented,(4) comparative and (5) forward oriented.

(1) Independent. The evaluation will be independent, carried out by entities and

persons free of the control of those responsible for the design and implementation of

ITC’s intervention. Independence implies freedom from political influence and

organizational pressures and incentives. It is characterized by full access to information

and by full autonomy in carrying out investigations and reporting findings4.

(2) Participatory. A participatory process will be facilitated and led by the Evaluation

Team. Key types of stakeholders at each level of implementation will be part of the

evaluation process at each phase: building the evaluation matrix, gathering data,

analysis and feedback. This participatory approach has been chosen to better learn and

capitalize on existing knowledge and to provide a greater base for the ownership of

recommendations.

(3) Output and Results Oriented: “Results” are defined as “outcomes, effects and

impacts” or all “changes induced by ITC’s activities”. Changes can be positive or

negative, primary or secondary, short or long term, resulting from a development

intervention5. Impact indicators are identified in the evaluation matrix. Measurements

refer to changes in beneficiaries’ capacities, which is the first level of impact, and

changes in end-users capacities, and in export performance and other development

concerns at the enterprise level, which are second level impacts. Assessment of results

will be done with the beneficiaries and end-users.

(4) Evaluation Criteria: The evaluation methodology will permit the assessment of ITC,

both in terms of its comparative advantage and its performance with regards to five of

the criteria generally adopted by all major donor agencies6:

4 OECD’s 2002 glossary

5 OECD / DAC glossary.

6 Definitions given here were extracted from: EU – Evaluation guidelines, UNDP HandBook on

Monitoring and Evaluating for results, DANIDA Evaluation guidelines. The criteria of impact won’t be



• Relevance: relates to a project/activity design and the extent to which its

stated objectives correctly address real needs and remain valid owing under

changing circumstances;

• Efficiency: relates to how well the various activities transform resources

(inputs) into outputs. The Budget Analysis will seek to benchmark cost per

type of deliverable and assess efficiency of ITC.

• Effectiveness: relates to the degree to which a project achieves its planned

results (goals, purposes and outputs) and how far were the project’s results

used by the beneficiaries (outreach). Cost-effectiveness is the relation

between the costs (inputs) and results produced by a project. A project is

more cost-effective when it achieves its results at the lowest possible cost

compared with alternative projects with the same intended results.

• Impact: relates to the longer term effects produced by the intervention,

whether positive or negative, directly or indirectly and intended or

unintended.

• Sustainability: Will the positive results of the project continue once the

project ends and what is needed to ensure continuity, and the multiplier

effect (replicability or scaling-up) to achieve wider impacts for the

region/nation as a whole.

In addition since ITC is working with or supporting networks both among countries and

within countries, we will also assess the outreach effects of ITC activities and networks.

This is an important aspect of output as noted above, but it is also linked to

sustainability.

(5) Forward Oriented: The analysis will provide a basis for identifying a range of

options for strengthening the comparative advantage of ITC, enhancing the results and

impact of its activities and developing the capacity of ITC and donor funding modalities.

Focus will be placed on matching the strengths of ITC and future opportunities.

used as stated in these sources, but is related to “effectiveness” in the proposed methodology for ITC’S

evaluation.



3.3 Overview of the Methodology

Evaluation Methodology at a Glance

External

Environment

of ITC

ExternalExternal

EnvironmentEnvironment

of ITCof ITCITCITCITC

Matching Strengths of ITC with

Opportunities in its environment

Matching Strengths of ITC with

Opportunities in its environment

RECOMMENDATIONSRECOMMENDATIONS

Information

collection

Information

collection

FACTUAL FINDINGSFACTUAL FINDINGS

Analysis and AssessmentsEfficiency

Effectiveness

Relevance

Impact

Sustainability

Analysis and AssessmentsEfficiency

Effectiveness

Relevance

Impact

Sustainability

Mapping

information

Mapping

information

Structured Reference

Framework (TRTA and ITC)

Structured Reference

Framework (TRTA and ITC)

The starting point is a structured

reference framework using the

Result-Based (RB) logical

framework applied to ITC as it

stands today and to Trade Related

Technical Assistance (TRTA) as a

benchmark. This reference

framework will be used to identify

indicators measuring each level of

the causality chain: resources,

outputs, and expected changes at

specific, corporate and ultimate

goal. It will also be used to

aggregate findings7, mapping key

information gathered through the

evaluation process.

A structured set of questions will be

used to collect information and

assess the external environment, and

ITC itself, first in the field then in

headquarters. This set of key

questions, and the RB reference

framework and linked indicators

together form the Evaluation

matrix.

The analysis of the performance of interventions of the ITC and of the external context

will be conducted according to the five criteria of evaluation and will use various

evaluative tools including interviews of beneficiaries and end-users, donors and other

international organisations, various survey techniques, and will be supplemented by the

budgetary analysis, the SWOT analysis and benchmarking.

The following diagram links the results chain and the evaluation criteria.

7 Factual statements about the project. (UNDP definition)


Final Inception Report v final pg 25/142 15/04/2005

Logical Framework Analysis Model

Evaluation Criteria LogFrame Levels

Means (inputs) (Materiel, personnel, financial resources)

Activities (process of converting inputs into results)

Results (confirmed planned deliverables

Purpose (benefits actually received)

Overall Objectives (overall lasting change, both at project / program

level and beyond

Efficiency links: from means through activities to results

Effectiveness links: (depend on assumptions made and on outside conditions)

Impact & Sustainability links: (depend on wide assumptions made and on outside conditions)

Relevance to the identified problems or real needs to be addressed

Design & Preparation


Version Final

26

Recommendations will build on matching strengths of ITC with opportunities in

its environment and lessons learned.

3.4 The Evaluation Matrix

3.4.1 TRTA Results-Based - reference framework

The results – based (RB) logical framework is used to build a structured reference

framework upon which ITC will be evaluated. See the diagram overleaf.


Version Final

27

TRTA corporate

Goal (impact)

Ultimate goal

MDG 8 ( impact)

TA/

Capacity

building

specific

Objectives

(outcomes)

Technical

assistance

Outputs

Inputs

Types of deliverables

1. Tools

2. Information sources

3. Advisory

4. Training services

5. Networking

6. Combination of above:

Programmes,initiatives

and projects

Funding:

- Level and sources of funds

- Beneficiary cost-sharing

- Rate of disbursements

Human resource

- Staff and consultants

- Skills and competences

- Business processes

TRTA’s results based – reference framework

Management process of outputs

- Requests

- Needs assessment

- Project design and fund raising

- Partnerships

- Implementation and procurement

- M&E

- Termination

Geographic focus (scale):

Global

Regional – inter, sub

Country

Target groups

(Outreach)

-Intermediaries

-End-users

Develop a global partnership for sustainable development and poverty alleviation,

through an open trading system that is rule-based and non-discriminatory

Enhance enterprise competitiveness

and national / sector export performance

Areas of technical competences transferred to

Trade support services (network and/or

private/ public institutions) and/ or end-users

1. Export strategy

2. Multilateral trading system (MTS) knowledge

and business implications

3. Enterprise management and competitiveness

4. Product development by sector

5. Market analysis and strategic market research

6. Public-private partnership and networking

7. Trade in services

8. Trade information management

9. E-competence

10. Trade finance

11. Standards and quality management ( including

TBT and SPS)

12. Supply chain management ( including public

procurement)

13. Export packaging

14. Legal aspects

15. Institutional strengthening of TSI

16. …

Channel of delivery: (3 tracks)

1. Global delivery of generic tools and programs

2. Integrated in multiagency, multi objectives programs

3. Regional/country specific pipeline of projects and tools

Technical competences available:

1-15 Areas of TRTA

a) – e) Areas of TA in development

Areas of technical

competence

linked to development

concerns:

a) Environmental

sustainability

b) Women in

development

c) Poverty eradication

d) Economic and

technical cooperation

among developing

countries

e) Digital divide

TRTA corporate

Goal (impact)

Ultimate goal

MDG 8 ( impact)

TA/

Capacity

building

specific

Objectives

(outcomes)

Technical

assistance

Outputs

Inputs

Types of deliverables

1. Tools

2. Information sources

3. Advisory

4. Training services

5. Networking

6. Combination of above:

Programmes,initiatives

and projects

Funding:

- Level and sources of funds

- Beneficiary cost-sharing

- Rate of disbursements

Human resource

- Staff and consultants

- Skills and competences

- Business processes

TRTA’s results based – reference framework


- Requests

- Needs assessment

- Project design and fund raising

- Partnerships

- Implementation and procurement

- M&E

- Termination

Geographic focus (scale):

Global

Regional – inter, sub

Country

Target groups

(Outreach)

-Intermediaries

-End-users

Develop a global partnership for sustainable development and poverty alleviation,

through an open trading system that is rule-based and non-discriminatory

Enhance enterprise competitiveness

and national / sector export performance

Areas of technical competences transferred to

Trade support services (network and/or

private/ public institutions) and/ or end-users

1. Export strategy


and business implications




6. Public-private partnership and networking



9. E-competence

10. Trade finance

11. Standards and quality management ( including

TBT and SPS)

12. Supply chain management ( including public

procurement)


14. Legal aspects


16. …

Channel of delivery: (3 tracks)

1. Global delivery of generic tools and programs

2. Integrated in multiagency, multi objectives programs

3. Regional/country specific pipeline of projects and tools

Technical competences available:

1-15 Areas of TRTA

a) – e) Areas of TA in development

Areas of technical

competence

linked to development

concerns:

a) Environmental

sustainability

b) Women in

development


d) Economic and

technical cooperation

among developing

countries

e) Digital divide


Version Final

28

During the desk study, corporate documents such as the Business Plan 2003-

2005, the Proposed Programme Budget for the biennium 2004 – 2005, the intranet

web sites, the 2003 Annual Report and the Organisation Chart have been

reviewed to get a clear view of what ITC is doing and why. Interviews with staff8

helped gain a clearer perspective of their activities. The OECD database on Trade

Related Technical Assistance (TRTA) initiated in 2001 has also been reviewed to

identify a common framework upon which ITC’s performance would be

compared to other similar service providers. A correspondence table9 of the

different structures shows the nuances and differences between these

perspectives. In light of the intricacies of the resulting table, it was decided to

break down ITC deliverables to their smallest component (6 types of

deliverables and 20 areas of competence, of which 15 related to trade and 5

related to development concerns). This places the focus on capacity building as

the key specific objective of any technical assistance program. It also helps

identify a corporate driving force to all trade related technical assistance (a

corporate goal of TRTA) and to build a link with the Millennium Development

Goals (MDG), particularly number 8, relating trade to sustainable development

and poverty alleviation, as the strategic or ultimate goal.

This RB reference framework will be used to benchmark ITC performance and

comparative advantage with other services providers. During the field study,

this reference framework will be used as an open tool to validate its structure

with other TRTA leaders, to better position ITC in this field of technical

assistance.

8 March 11 and 12th, with Division Directors and sections chiefs from the Divisions of Product

and Market Development (DPMD), Division of Trade Support Services (DTSS), and the Division

of Technical Co-operation Co-ordination (DTCC). Separate meetings were also held with the

Deputy Executive Director and the Director of the Division on Programme Support on budgetary

and organisational activities, with the Co-ordinator of the Executive Forum, and with the Senior

Officer for Evaluation. On March 16, a draft of the Evaluation Matrix was discussed in a

workshop with almost all the Management and Senior Staff of the ITC. Several meetings were

held with the Senior Management Committee of the ITC. 9 Annex. Correspondence table for key technical areas of competences of ITC, showing the

various ways to articulate what ITC is doing. This tool was used to help clarify and simplify the

logic of intervention of TRTA and ITC in TRTA. During the field study, the TRTA RB reference

framework will be confronted to other service providers to assess its relevance.


Version Final

29

The following five distinct elements within this RB reference framework have

been agreed upon:

(1) Deliverable: A deliverable is an ITC output, a service delivered by ITC. A

deliverable can be any combination of types of deliverable + areas of competence.

(2) Types of deliverables:

Defined as different types of support used to transfer competence and build

capacity:

• Tools: to support and to sustain the delivery of TRTA through partner

institutions; they include practical guides, either generic or adapted to local

requirements, methodologies and approaches, benchmarking & assessment

tools.

• Information sources: to give access to information on international trade and

business development through internet, CDROM or paper form: include web

sites, newsletters, bulletins, press abstracts, online information systems,

Trade MAP, country market analysis profile (the country MAP), Market

access map (MacMap), Product market analysis portal (Product Map),

databases, e-shop.

• Advisory: to provide technical expertise to build awareness, to carry out

needs assessments, to solve a problem, to study an issue, etc.

• Training services and modules: as referred to here, will be limited to

training modules and formal training, since training, as generally

understood, underlies any activity carried out in the technical assistance –

capacity building sector

• Networks: defined as visible networks, and includes conferences, exhibitions

like the tools of trade fairs, the Africa meet, etc.

• Programmes, initiatives and projects: incorporates a certain number of

proven tools and technical competencies under a convention, with a

perceptible time-proven impact. These programmes can be at preparation or

implementing phase

Source of information on deliverables:

In the Project Portal, the main source of information on ITC’s deliverables in a

country (2001-2004), each entry is either a project/programme or an activity

carried by an ITC division / section targeting that country. In the name of the

entry or in the brief description attached to it, we identify the types of deliverable

and areas of technical competencies. Each entry corresponds to a product

delivered at the country level. The list of projects in the sample of countries will

be completed for 5 years.


Version Final

30

The total amount identified in the Project’s portal database corresponds to the

amount of the project, not the amount dedicated specifically to the country.

Small amounts could then be an indication of an amount specifically invested in

one country. Negligible amount - 0 – can be found and should be assessed from

a country’s perspective.

The e-shop provides a list of all material (tools, information sources, training

material) published either on paper or e- format.

Advisory services and visible networks will have to be identified through other

means.

(3) Areas of technical competence:

Defined as a technical aspect of trade, for which a certain level of knowledge is to

be reached, and a certain number of skills10 is to be adopted depending of the

targeted audience. It is an open list for the reference framework to include areas

identified with other similar service providers during the field study.

There are 15 areas of technical competence in ITC.

1. Export strategy





6. Public – private partnership and networking



9. E-competence

10. Trade finance

11. Standards and quality management (including Technical Barriers to

Trade (TBT) and Sanitary and Phytosanitary measures (SPS))

12. Supply chain management (including public procurement)


14. Legal aspects


A sub-group of 5 is to be added related to issues of common concern and the

Millennium Development Goals. These are cross cutting technical areas linked to

development concerns. Any trade technical competence can be linked with one

or more development area.

a) Environmental sustainability

10 See indicators of capacity building for more details.


Version Final

31

b) Women in development


d) Economic and technical cooperation among developing countries

e) Digital divide

The areas of technical competence will be assessed in two ways: as an input,

when looked at their availability in ITC staff and consultants, and as an outcome,

or a change induced by ITC’s activity, on targeted groups.

(4) Target groups: Trade Support Institutions (TSIs) are the directly targeted

group of ITC’s activity. These TSIs can be public or private service providers to

businesses. The businesses are the end-users of ITC’s services. The outreach of

ITC will be measured in terms of number of TSIs for the first level of impact, the

capacity building specific objective. The multiplier-effect will be measured in the

number of end-users reached by TSIs benefiting from ITC’s support.

TSIs can be characterised by the following criteria:

- Private or public,

- Sector-specific or general

- Main area of competency,

- Centralised, decentralised or both.

- Number of years in TRTA business

End-user beneficiaries of TSI services can be distinguished as follows:

- SME or large ( threshold for SMEs to be agreed upon)

- sector of activity

- Prospective or actual exporters

- Number of employees – women ratio and gender sensitivity

- Centralised or decentralised location in country

(5) Channel of delivery (or Tracks): there are 3 channels for delivering ITC’s

activities, also referred to as the 3 tracks:

• Track 1: Global delivery of generic tools or programs developed through

headquarters – initiated research and under its product-network approach.

(This track will include for example: development of a new programme like

Trade-in Services )

• Track 2: ITC participation in multi-agency program, through which ITC

provides specialised inputs related to Trade to a more broad based

cooperative initiative. (Importance will continue to be given to increasing

field activity through this track.)

• Track 3: Pipeline of national and sub-regional projects that concentrates on

the two-fold objectives of identifying export opportunities for particular


Version Final

32

product sectors and strengthening the national trade support network upon

which exploitation of these opportunities by the business sector relies. (ITC’s

first priority).

The distinction between track 1 and 3 is not always clear-cut. Track 1 might on

occasion be said to be the development stage including the test in the field, using

ITC’s own funds and global trust fund, whereas track 3 could be viewed as more

country or region adapted and funded by donors bilateral flagged funds.

Geographic focus (scale): There are 3 scales for geographic coverage of the

product delivered to the country. These are:

• Global: an access of the country to a generic product;

• Regional: an access of the country to a product that has been designed

for the region, sub-region, or inter-region specificities; and

• Country: an access of the country to a product that as been adapted to

its specific local requirements.

3.4.2 ITC - Results-Based logic of intervention

The ITC results-based logic of intervention is built around ITC’s mission, both at

the level of strategic goals, and at the level of the five (5) corporate goals, which

specify the types of changes ITC aims to achieve by delivering TRTA. The

evaluation will assess the relevance of these statements and their targets.

The ITC deliverables are structured in “programmes”, expanding from the 7

initial programmes to 17, with new ones being designed to answer evolving

requests. These programmes can be assimilated to the 15 areas of technical

competence.


Version Final

33

ITC’s

corporate

Goals

Mission

Technical

assistance

Outputs

Inputs Funding

Humanresource

ITC’sResultsBased– logicof intervention


To support developing and transition economies, and particularly

their business sector, in their efforts to realise their full potential for

developing exports and improving import operations

with the ultimate goal of achieving sustainable development

1. To facilitate the

integrationof

enterprises

into theMTS

2. To support national

efforts to design

and implement

trade development

strategies

3. To strenghten

key trade support

services, both

public and private

4. To improve

export performance

in sectors of critical

importance and

opportunity

5. To foster international

competitivenesswithin

thebusiness communnity

as a whole and the small

andmedium-sized

enterprise (SME) sector

in particular7 Programmes + extras:

1. Strategic and operational market research (MAS)

2. Business advisory services (BAS)


4. Export training capacity (EMDS)

5. Sector specific product development (MDS)


7. International purchasingandsupply chainmanagement (IPSMS)

8. Export LedPovertyReduction (EPRP)

9. E-TradeBridge Program

10. SouthSouthTradePromotion (SSTP)

11. Joint IntegratedTechnical Assistance Program(JITAP)

12. IntegratedFramework (IF)

13. World TradeNet (andbusiness for development) (WTN)

14. ExecutiveForum

15. CapacityBuilding andNetworkingBusiness Information Services (CAPNET/BIS)

16. Programfor Building AfricanCapacity for Trade (PACT)

17. Programfor Competitiveness Improvement of SMEs (PROCIP)

18. …

Types of deliverables:

Advisory and training services

Information sources

Tools andproducts

Global programs

Channel of

delivery

(3 tracks)

3.4.3 The key indicators

Based on this analysis of the ITC Results – Based logic of intervention, the TRTA

Results-Based reference framework clarifies what TRTA is about and what

objectives it is aiming at, with ITC’s specificities. These structured reference

frameworks are used to identify indicators for each level of the results chain:

strategic goal, corporate goals, technical assistance specific objectives, outputs

and inputs.


Version Final

34

Types of indicators:

Inputs:

Input Indicators are standard among many types of organisations. They are

related to financial resources, human resources, management processes and time

frames.

Outputs:

Output Indicators are common among many organisations but they are not

always measured as carefully and comprehensively as inputs. They are

measured in terms of volume, quality and outreach.

Impacts:

Technical assistance11 specific objectives are the first level of ITC’s impact, with

the measurement of changes directly linked to ITC’s activities. These can be

measured and assessed with the direct beneficiaries.

Corporate and strategic goals are the second level of impact, or the rolling effect

induced by the first changes identified. Measurements can be done at national

scale in circumstances where sufficient time has elapsed and multiplier effects

may have occurred. Measurement will be done in this evaluation at the scale of

end-users or enterprises, which directly benefited from the support of a TSI

linked to ITC’s activities. (These measurements are used at this second level of

change to guide decision-making in results-based management). These

measurements can also be compared to trends in national data and analysed

with external factors to build synergy among stakeholders as the achievements of

these goals are outside the capability of any single organisation’s action.

Issues related to indicators:

TRTA impact measurement is a relatively new field of research, which the

development community has only emphasized in recent years. The identification

of indicators is certainly the most difficult part of any evaluation process. The

indicators identified in the matrix in Annex 3, have been gathered through a

review of corporate documents, project documents or during discussion with

11 Capacity development (UNDP definition): The process by which individuals, groups,

organizations and countries develop, enhance and organize their systems, resources and

knowledge—all reflected in their abilities (individually and collectively) to perform

functions, solve problems and set and achieve objectives. Capacity development is also

referred to as capacity building or strengthening.


Version Final

35

ITC’s staff and review of the available literature. It is too early to build a

consensus on these indicators, as no measurement is yet available to assess their

relevance or relative significance.

The following guidelines were relied upon to help sort out the indicators from

many sources of information used by managers.

1. Indicators measuring ITC activities outputs and their outreach are relatively

straightforward to identify as they refer to tangible services delivered to

identifiable targeted audiences.

2. Results and impact Indicators should be:

• Linked to the problem that needed solving or to the success expected (within

a certain time frame in 3-5 years)

• At the scale of the basic unit of information (direct beneficiary and end-user)

for later aggregation

• Available for measurement by field stakeholder(s)

• Designed with a clear methodology of measurement

• Included in a comparative table (either before and after, actual and

normative, etc.), and

• Most important, assessed as to the relative importance and relevance of the

indicators in the decision making/ action process.

The matrix presented in the annex summarises the indicators identified for each

level of the Results-Based structure, with the type of data (quantitative and

qualitative) that will be used. (When targets were available in ITC’s documents,

they are specified and will be used during the evaluation process as one

benchmark). As there are often no baseline data, the actual status of impact

indicators will be measured, and discussed with beneficiaries against their

assessment of the situation 5 years before and 5 years in the future.

Impact measurements of other TRTA services providers will also be reviewed, to

enhance our overall understanding of the current state of the art of TRTA impact

measurement. We have identified priority indicators and we have developed

detailed questionnaires and tools for the country-level field studies in order to

make operational and to set priorities among the large number of potential

indicators. At the end of the evaluation process, these key indicators will be

reviewed to determine if these were most useful for the assessment of the

performance of ITC will be identified.

The key indicators are summarised as follow.


Version Final

36

ITC key indicators to be used for the evaluation:

For the ultimate goal of sustainable development and a global partnership for an

open trading system:

1. Awareness of the relevance of trade performance to overall development

2. Improved employment and region / poor outreach linked to enterprise

export expansion

3. Countries advocacy network’s capacity and effectiveness in building a rules-

based and non discriminatory MTS

For the corporate goal in TRTA, enhancing trade performance and

competitiveness:

1. Trade performance at country, sector and enterprise scale

2. Share of ITC in TRTA at country scale

At the level of capacity building :

1. Multiplier effect of intermediaries on the number of exporting enterprises,

particularly SMEs, and correlated impact on women

2. Changes in trade support services profile at national network scale and

individual TSI scale

3. TSI perception on changes in their capacity in respect to an ideal profile

identified as follows:

- Technical skills (one of the 15 +5 areas of competencies)

- Capacity to transfer knowledge

- Needs assessments / advocacy and diagnostic

- Project management, internal coordination and follow-up

- Promotion to the client and to the donor

- Networking

4. Current state of technical competence in country with respect to the future

foreseen.

5. Development of networks for exchange of information and best practice

techniques.

At the level of ITC’s outputs:

1. ITC’s portfolio of products

2. Uniqueness of ITC deliverable compared to its competitors

3. Demand: TSI needs

4. Demand: End-user needs

5. “Competitors”


Version Final

37

ITC’s outreach to target groups:

1. Access of ITC to intermediaries

2. Intermediaries’ awareness of ITC and access to deliverables

3. Outreach

4. Price of services – willingness of intermediaries to pay for a deliverable

And finally at the level of inputs: indicators will be on

1. financial resources

2. human resources, and

3. management processes

3.4.4 The key evaluation questions

A list of key evaluation questions has been put together to structure and

sequence the process of gathering information. This master questionnaire

is presented in the annex.

It has the following structure.

THEMES OF KEY EVALUATION QUESTIONS

ITC’s external environment

• Global enabling environment

• Enabling environment of beneficiary country

• Changing needs of the beneficiary countries

• Overall availability of services similar to ITC – TRTA

• Emerging issues

ITC’s intervention

• Specification of portfolio of ITC’s deliverables

• Outputs

• ITC target groups

• Geographical focus of ITC

• Impact of ITC interventions

• Performance of ITC deliverables

• Identification and assessment of other providers

Managerial challenges of ITC

• Strategic coherence between interventions, policies and objectives


Version Final

38

• Strategic planning

• Promotion capacity

• Networking and partnership capacity

• Quality issues

Functional issues

• Operations

• Product development

• Human resource management

• Financial resource management

• Monitors and control systems

• Procurement

Organisational capability

• Technical skills and competencies

• Project cycle management including monitoring and evaluation

• Transfer knowledge and capacity building

• E-competence

It takes into account evaluative questions identified in the terms of reference and

other questions raised during the desk review.

It identifies where information can be gathered: documentation review,

interviews with ITC staff, TSI and end-users in the field, other stakeholders at a

global scale and thematic studies.

It will be broken down into specific questionnaires for different types of

stakeholders. Findings will be summarised in tables according to the indicators.

3.5 Implementation Modalities

The following are the four implementation modalities for the evaluation. The

first evaluation modality involves an assessment of the external environment, the

relative role of different international organisations and partners, the perspective

of donors and the current approaches to trade development. The second

evaluation modality is an evaluation of ITC performance and comparative

advantage at the country level. The third evaluation modality is an evaluation of

the main ITC products or competencies. The fourth evaluation modality is an

evaluation of the corporate organisation and management and the budgetary

analysis.


Version Final

39

We use these four modalities in order to evaluate the wide range of ITC

competencies, the full scope of ITC deliverables and the three tracks of delivery,

global, multi-agency and regional/country level. We also propose to use four

modalities in evaluation in order to provide independent sources of data and

information which test, validate and triangulate the information and findings

obtained from the other modalities.

3.5.1 Assessment of ITC relative position in TRTA and of

Developed Country Donor/TSI Perspectives

The first evaluation modality is the assessment of TRTA performance and ITC

added value from the perspective of donors at headquarters level and developed

countries’ TSIs. This analysis focuses on the external context of the ITC and

builds on the sections of the Desk Study related to the external context. The ITC

is compared to other international organisations in terms of its perceived

strengths and weaknesses. Main issues include:

• How is TRTA evolving? Past, today and future? And where does ITC stand

in TRTA?

• How do donors perceive ITC?

• What are the perceived strengths and weaknesses of ITC?

• How do ITC products compare with the products of other international

organisations?

• When and how does ITC partner with other international organizations? Is

there overlap? What is the comparative advantage of ITC? Where and when

are their complementarities?

• What are the current approaches to the management and dissemination of

trade information and to trade development in developed countries?

There will be technical study examining the perspectives of key donors and

potential partner/rival international organizations on the strengths and

weaknesses of the ITC and its perceived comparative advantage. A selected

number of international organisations including WTO, UNCTAD, ISO, ITU,

UNIDO, OAS, Commonwealth Secretariat, WIPO, and IFIS will be interviewed

or surveyed to determine their detailed work programs and potential

complementarities. In addition a selected number of donor countries will be

interviewed and/or receive questionnaires and the views of NGOs will be sought

and their input encouraged.

There will be a technical study examining the perspective of developed country

TPOs on the management of trade information and assessing the current

models for trade development, and the effectiveness of trade development

approaches for developing countries. ITC services are predominantly provided


Version Final

40

through the medium of developing country TSIs, most often the country’s

national TPO. The calibre and capability of these TSIs have a crucial bearing on

the overall effectiveness of ITC interventions. While we cannot evaluate these

institutions comprehensively, we will combine the field interviews of TPOs/TSIs

(under 3.5.2 below) with a selected number of interviews with European TPOs to

assess the suitability and appropriateness of the TPO delivery model in

developing countries.

3.5.2 Assessment of ITC performance and comparative

advantage at the Country or Regional Level

The second modality for implementation of the evaluation is the evaluation from

the individual Country perspective of ITC’s performance, and its comparative

advantage (vis a vis other TRTA suppliers) at the country or regional level. This

country-level evaluation will be carried out in 10 countries, (see discussion of

country selection in the Annex), the results of which will permit comparison:

• Of the ITC portfolio of products (in terms of the frequency of use of different

ones, and their effectiveness); highlighting the limiting factors of human,

institutional and financial constraints

• Of results between LDC, OLIC, LMIC and CEE/CIS countries, considering

the importance of country-specific environmental and structural factors;

• Between regions and countries: importance of external factors. Some

perspectives will also be available through this study:

• On the channel of delivery integrated in multi-agencies and multi-objectives

programmes. (track 2) ( for example in Burkina Faso, Tanzania, Kenya); and

• Of the performance at the country level of the horizontal product areas in the

ITC evaluation as discussed in the next section.

The country level evaluations will be conducted on the basis of the Country

Evaluation tools developed detailed questionnaires and data sheets for the Core

Evaluation Team Member and the Local/Regional Consultant. The CE tools and

data sheets are discussed below.

3.5.3 Assessment of ITC Products

The third implementation modality is an assessment of the ITC performance

based on horizontal or functional analysis of the technical capacity and

competence of ITC focusing on selected services and products of ITC. Since

much of ITC services are provided through Track 1, the global track, it is

necessary to evaluate these products or services at headquarters and to assess the

products and their dissemination and use also for Track 2, through working with


Version Final

41

other agencies. For this analysis the products and services of ITC will be grouped

into the following clusters:

� Product and Market Development, Strategic Market Analysis, Trade

Information Management and Trade in Services comprise four of the seven

programmes of the ITC. For these programmes or services a series of specific

e-questionnaires will be utilised in order to assess their utilisation by

Beneficiaries and End-Users. The products and services will be compared to,

and bench marked against, other comparable products and services through

various techniques including expert peer review. Main questions are related

to: who are the users, what is the quality and added value of these products

for the beneficiaries, compared to similar providers.

• Participation in the Trading System, Business Framework, National Export

Strategies and Networks comprise another important cluster of products. The

ITC has two complementary roles of supporting business dialogue and

networks about the trading regime and of supporting business enterprises in

developing countries to be able to function effectively in international

commerce. The evaluation of these products will examine the quality and

utility of the ITC tools and advisory services in these areas, the effectiveness

of ITC in developing networks and the effectiveness of dialogue and

information exchange within these networks. The business advocacy

capacity and effective integration is evaluated based on the Business for

Development Forum held in Rio de Janeiro in advance of the UNCTAD 11

meeting in Brazil in June. Specific ITC competences to be evaluated include

competences (1) export strategy, (2) Multilateral Trading System, (11)

standards and quality, (12) import purchasing and supply chain

management, (13) export packaging, and (14) legal framework. The

complementarities with, and effectiveness relative to, other services

providers and international organizations will be examined.

• The SME perspective on ITC performance and comparative advantage, in the

E-trade bridge program; assessing the effectiveness in supporting enterprise

competitiveness and capacity building. In assessing Enterprise

competitiveness, the main focus is on: role, capacity and interest of TSI in

transferring knowledge to SME, and capacity building of TSI brought by ITC.

• The integration of development concerns in TRTA: lessons of success in

Export – led poverty reduction and in developing South-South Trade, with

key issues being views on links between trade and development, and ITC

competences in those fields. These projects will be evaluated based on field

work in South Africa and in China. Other development concerns will be

looked at in the country studies.

In addition to examination of these products and services at the country level,

these products will be evaluated directly through various techniques including

the evaluation of particular activities such as the Business for Development


Version Final

42

Forum; through evaluation of the utilization of the ITC web site or publications;

through the use of e-questionnaires and surveys to contact participants in

networks and to survey end-users and beneficiaries; and through the use of

expert peer review of particular tools and products.

3.5.4 Assessment of Corporate Organisation, Financing and

Management

The fourth implementation modality will be an assessment of the corporate

organisation, the financial constraints, the cost effectiveness of services and

products, and the management capacity. The main issues include:

• Project management cycle of outputs: the “project” concept in ITC, the

process of project identification, the internal and external coordination for

leverage, the internal quality control process, the effectiveness of monitoring

of project implementation, the internal evaluation process and learning

capacity, the accuracy and validity of the needs assessment, the continuity

and feed-back flows between ITC and field (capacity building is a long term

type of impact, while ITC’s time frame is short term), the degree of result-

based management implementation, and the management information

system (MIS) that supports the PCM and information needs for management.

• Financial resources of ITC and budgetary analysis: Predictability of funding

and project development and selection, implications of funding uncertainty

from one year to the next, donors funding mechanisms and constraints,

adequacy of financing to meet demand, cash-flow and disbursement

considerations, relative costs of services and products, allocation of

overheads, comparative costs for comparable organisations, impact of scale

of the organisation on costs, and pricing of, and willingness to pay for ITC

products. Financial allocation ratio for development of products and

technical assistance (Total budget – total costs project portal per section),

funding ratio per donor, funding ratio per section, trends, level and use of

income generating products (“revolving funds”), pricing policy of ITC

products and contribution of beneficiaries (willingness to pay) . The analysis

of financial issues will also consider governance issues for the ITC.

• Organisational structure and effectiveness, particularly the procurement

process, the human resource allocation (the ratio of staff to key outputs

which is linked to the budgetary analysis), the governance of ITC (external

and internal), the e-orientation and e-readiness of the organisation and the

use of the results of evaluation and performance appraisal at the

organisational level will be evaluated.


Version Final

43

3.6 Data gathering tools

The main tools that will be used to gather information are:

a) Documentation review

b) Budget analysis

c) Rapid appraisal tools with stakeholders (semi structured interviews,

country assessment workshop, corporate workshop, focus group, …) and

data sheets to facilitate data processing during and after field studies.

d) Surveys

e) Direct observation

f) Expert and peer review

g) Ranking

h) SWOT tool

a) Documentation review:

The documentation review is very important to identify:

• Objectives and measures or examples of achievements

• Beneficiaries ( intended and effective)

• Activities,

• Geographic coverage

• Services providers

• Cost and sources of funds

• Success and failures

b) Budget analysis:

ITC is a relatively small organisation, but the budgetary process is complex. The

regular budget comes half from the United Nations and half from the World

Trade Organization. In previous decades, operations funds came primarily from

UNDP, but this source of funding has declined and bilateral donors have become

the major sources of operating funding. A careful analysis of the budget

information – to the extent that detailed information is available – will be made

for identifying factual information and analysing strengths and weaknesses of

the financial resource issues in ITC. Specific issues with respect to donor funding

modalities, such as availability of multi-year funding and the implications for

ITC planning and activities will be examined.

The budget analysis will examine the revenue generation aspects of ITC products

and services and consider the pricing strategies. The analysis will also examine

the potential willingness to pay for products and services, not necessarily as a

revenue generation strategy, but as measures of potential benefits and impacts.


Version Final

44

c) Rapid appraisal tools (semi structured interviews, country assessment

workshop, corporate workshop, focus groups)

Semi-structured interviews will be the most common tool used to gather

information. These semi-structured interviews are most useful for developing an

in depth understanding of qualitative issues and to understand why and how

something happens, motivation, behaviour and perspectives of customers and

partners, to see how attitude can help explain successes and shortcomings, to

frame the issues.

Consistency among local consultants and core evaluators will be ensured by the

detailed methodology and tool kit provided to each of them. Regional launching

workshops will be held before starting the field study phase.

Tool kit of information for local consultant to gather:

Country data sheet

Sector data sheet if ITC has a sector programme

Detailed questionnaires

Donor data sheet

Project data sheet

TSI data sheet

Tool kit of information to core evaluators to synthesise findings:

Country evaluation data sheet

Interview data sheet

Workshop data sheet

Direct observation data sheet

Semi-structured interviews: will be carried out either individually or as focus

groups, by the local consultant and cross checked by the evaluator. They will be

carried with

• TSIs, TPO and other associations related to the project

• Other TSIs not involved with the project

• Donors

• End-users

• Other international organisations

A country workshop, facilitated by the core evaluator, and where main

stakeholders will be participating, will assess findings with respects to the 4

criteria of the evaluation, identify strengths and opportunities, and prepare

formulation of recommendations.


Version Final

45

A corporate workshop with the ITC will be facilitated during the last phase of

the evaluation (the final report preparation), by the evaluation team, with the

same purpose.

By combining multiple data sources, methods, analysis, evaluators seek to

overcome the bias that comes from single informant, single method, single

observer or single theory studies. The two workshops are a way to validate

findings through triangulation and transparency.

d) Surveys

Survey techniques and questionnaires will be used at both the country level and

at the horizontal/global level. The website will include an e-questionnaire and

direct surveys will be conducted of selected target groups for selected ITC

services and products. Certainly users of the ITC website will be surveyed, but

other target groups will be surveyed as a control.

The detailed questionnaires for the country studies and the horizontal/global

technical studies will be derived based on the framework of the evaluative

questions outlined in Annex 1.

e) Direct observation

A simple method: observation, made more effective by structuring its use. Much

can be learned by watching what people actually do. It is useful when:

• Performance indicators indicate results are not being accomplished as

planned

• Implementation problems are suspected but not understood

• Details of an activity need to be assessed, such as whether tasks are being

implemented to standards.

During field study phase, direct observation will be useful for certain activities

such as:

• Tools fair

• TSI network meeting

• Business for development in UNCTAD XI.

• TSI providing support to enterprises

f) Expert and peer review


Version Final

46

Selective use will be made of expert and specialist peer review of selected ITC

services and products.

g) Ranking

Ranking is critical when comparing elements of information on the basis of

agreed criteria. Besides the resulting matrix, one other great value of ranking

comes from the discussions that are provoked as participants come to a decision

about the final score of each option as well as on setting the criteria. In the

discussion, the reasons for preferences and rejections of options emerge.

This method will be used with small or large groups and can take from 15

minutes to one hour depending on the size of the group and how much debate

there is over the initial statement. The final outcome is a set of independent

judgments made in a non-threatening private way that will allow people to

generate a group judgment without social pressure to conform.

It will be used to measure qualitative data for impact indicators during field

study and to assess evaluation criteria during country workshop.

h) SWOT tool

The aim of the SWOT analysis tool (Strengths, Weaknesses, Opportunities and

Threats) is to capture a snapshot of the main strengths and weaknesses of an

organisation and the opportunities and threats which may affect the way in

which it works in the future. The SWOT analysis will utilise interviews at the

country level and on both interviews and expert peer review at the global level.

A structured assessment of the fit between what the organisation can (strengths)

and cannot (weaknesses) presently do, and the environmental conditions

working for (opportunities) and against (threats) ITC will be carried out.

3.6.1 Sampling

The detailed sampling approaches for the country studies are outlined in Annex

3 below. The sample selected for country studies is heavily weighted toward

countries where ITC has projects and interventions, both in global activities and

in country specific projects. The assessment of the ITC products will provide a

cross-check on the country level interventions.


Version Final

47

3.6.2 Evaluation steps

There are four (4) phases in the evaluation process, with overlapping between

each phase due to the participatory and adaptive approach used.

Inception phase

The purpose of this phase is to have an overview of ITC and the detailed

methodology that will be used for its evaluation. A draft inception report was

submitted to the Evaluation Management team, providing an overview of ITC,

the evaluation mandate and a draft methodology. The final inception report

finalises the evaluation methodology based on detailed information gathered

through the desk review and a corporate workshop with ITC’s staff to validate

the result-based reference framework and discuss the evaluation matrix. The

outline of the inception report is:

• Background and rationale for the evaluation

• Methodology of the evaluation

• Communication strategies

• Quality assurance methodology

• Risks, mitigating strategies and assumptions

• Project planning.

Desk review phase

The desk review phase consists in documentation review and ITC’s staff

meetings to get a clear view of what ITC is doing and aiming at, and what are the

issues to look at more deeply during the field study. During this phase, country

sampling has been made based on ITC’s portfolio of interventions, background

information has been collected, and evaluation tools have been finalised. A desk

review report has been prepared to present preliminary results on:

• Global context of Trade related technical assistance

• ITC in the international system

• ITC structure, activities and budget

• Summary of previous evaluations

• Preparation of field studies

• Documentation and sources

Field and technical study phase

During the field study phase, documentation review will continue, more

specifically focussed on documents related to the list of field studies. A regional

workshop will launch the field studies in the countries, with the purpose to train

the local consultants on the evaluation methods and tools and to provide

consistency to the work carried by more than 20 people. A typical country study

will consist in data gathering and contacts/ meetings conducted by the local

consultant, followed by a core evaluator visit to cross-check data gathered, carry


Version Final

48

semi-structured interviews, synthesise findings and facilitate a country

workshop to assess with stakeholders, findings with respects to the 4 evaluation

criteria and strengths and weaknesses.

A field study report with the purpose to present aggregated findings will

complete the field phase.

Synthesis phase

During the last phase, aggregated findings will be assessed at the corporate level

with data gathering on corporate issues, focus group and corporate workshop to

assess ITC with respects to the five evaluation criteria and strengths and

weaknesses, and identify recommendations based on the match between

strengths and opportunities found in the external environment.

The final report will present the evaluation results (findings and assessments)

and recommendations.

3.7 Approach to the Evaluation

The evaluation will be independent, carried out by entities and persons free of

the control of those responsible for the design and implementation of the

development intervention. Independence implies freedom from political

influence and organizational pressure. It is characterized by full access to

information and by full autonomy in carrying out investigations and reporting

findings. ( OECD’s 2002 glossary)

The evaluation methodology’s approach is based on four characteristics:

1. It will be participatory. A participatory process will be facilitated and led

by the Evaluation Team, an independent body. All types of stakeholders

at each level of implementation will be part of the evaluation process at

each phase: building the evaluation matrix, gathering data, analysing and

feedback. This participatory approach has been chosen to better learn

and capitalize on existing knowledge and to provide a greater base for

recommendations’ ownership.

2. It will be results oriented. Results include outcomes, effects and impacts.

Impacts are positive and negative, primary and secondary long term

effects produced by a development intervention, directly or indirectly,


Version Final

49

intended or unintended.(OECD’s glossary). Results and impact

measurements will be identified in the evaluation matrix, for the three

levels of objectives: strategic, corporate and specific. They will measure

changes induced by ITC’s outputs. Measurements will be about change in

behaviour and effects on trade and development.

3. It will build on a Bottom – up approach, based on outputs delivered by

ITC, and the responses of direct clients and end-users. Information will be

gathered at this specific scale and then aggregated for analysis at country,

regional and world wide scale.

4. It will also utilise a Top-down approach based on the SWOT Analysis

and the Budgetary Analysis to assess ITC’s objectives in the changing

global context and the impact of funding mechanisms on effective

delivery of programmes.

There might be a need to give consideration to interregional or sub-regional

levels. The desk review will help the evaluation team to determine the level of

attention needed for this specific level.

3.8 Communications Strategies

The overall approach to communications will be to provide information on the

evaluation to all potentially interested parties and to provide for opportunities

for input into the evaluation.

3.8.1 Communication with Stakeholders

The Communication with Stakeholders will occur formally through meetings

with, and reporting to, the Management Group and the Core Group constituted

to supervise and to advise the evaluation. Of course information will be made

available to the Joint Advisory Group JAG and specific reporting to the JAG is

planned. In addition to this formal channel, informal contact will be maintained

with Geneva-based Delegates from developing, transition and donor countries.

Broader contacts will be maintained with business associations and NGOs

involved in trade and development issues.

3.8.2 Public Communications and Website

There are two facets of public communications. One is media activities. Another

is the development of a web site.


Version Final

50

Public Communications – Media Activities

Target Audience

The objective of any communications activity is twofold:

1. first to communicate a specific message from the sender to a particular target

audience, and,

2. second, to raise awareness among secondary target audience that is closely

allied with the target audience, but is not itself a target for the message.

In the case of the ITC Evaluation, there are a number of target audiences to be

addressed. One of the main ones is composed of the various recipients of ITC

assistance over the years, as well as their clients, e.g. Trade Promotion Offices

(TPO) and their clients (exporters, importers, business associations and the like).

A second target audience is made up of the various donors and partners of the

ITC as well as the ITC itself. This includes the organizations funding the ITC

evaluation.

The secondary target audience for this project is made up of other stakeholders

in the development arena such as NGOs, national and regional governments,

potential users of ITC services, etc.

Communications Mechanisms

Given that there is a substantial amount of “noise” in all communications

channels, and given that an activity such as evaluation of the ITC is not

particularly newsworthy for media that are more attuned to disasters (natural

and otherwise), wars, and human tragedies of one kind or another, we propose

to provide information on the project in a very factual and straightforward

manner to the various target audiences.

We propose to use three communications mechanisms – two active and one

passive.

1. The first active mechanism will be direct communications with recipients

of ITC assistance. This will be in the form of e-mail, fax or paper-based

mail describing the evaluation and its progress. Certain recipients will

also be advised that they may be visited by members of the evaluation

team. Recipients will also be told that a web page is available for them to

use to monitor project progress and to allow for their participation in

project activities.

2. The second active form of communication will include circulation of brief

descriptions of activities and results of the evaluation through selected

channels such as the ITC Forum and newsletters of organisations and

NGOS.


Version Final

51

3. The more passive mechanism is the web site itself which is described in

the next section of this report. In addition to providing project

information, the web site will be used for conducting surveys of various

kinds and will be a feedback mechanism whereby the project team can

obtain information from a variety of audiences.

Copies of reports and executive summaries also will be available. The web site

will have both full copies and executive summaries of reports available for

downloading once the evaluation management committee has approved their

release.


Version Final

52

Summary – Communications Activities Information Available Target Audience Mechanism Used

Project Reports All All with website the

preferred mechanism

Project Executive

Summaries

All All with website the

preferred mechanism

Press releases Media (primarily), ITC,

its partners and

stakeholders

Newletters, website

Project progress

(activities)

Beneficiaries, followed by

ITC and its stakeholders

Website

Public Communications – website

A public-access website will be developed as a part of the project’s

communications strategy. The main objective of the website will be to ensure

transparency of the evaluation process and its results. A corollary objective is,

through the application of interactive evaluation tools, to involve a broader

audience of ITC clients, supporters and other stakeholders in the evaluation of

the ITC. A third objective is to allow any individual or organization that wishes

to, to provide feedback and comments on any component of the evaluation or of

ITC’s activities.

The main information the website will contain is as follows:

1. Information about the evaluation project (Box 1):

- Background/key issues/why the evaluation is being

undertaken

- The main objectives of the ITC evaluation/key evaluation

questions;

- The project duration and key milestones;

• Terms of reference for the evaluation:

- Appendix A of the Contract;

• Project Donors:

- Information on who the project donors are;

- Description of their role and involvement in the

evaluation/ITC work;

- project implementation monitoring bodies

• Links:

- to donor websites, such as DANIDA; WTO, UNCTAD,

UNDP, etc.

- To ITC


Version Final

53

- to beneficiary websites and previous project websites

2. Consortium presentation (Box 3)

- Names of the consortium partners

- Presentation of each consortium partner

- Links to websites of the consortium partners

3. Presentation of the Team (Box 2):

- Information on the evaluation team members

4. Project documents (Box 4):

- Information on the expected evaluation

outputs/documents

- Description of the evaluation methodology and main

activities to achieve the project objectives;

- Executive summaries12 and full texts of project reports

- Final Report

- Press releases

5. Feedback (Box 5)

6. Project working papers (Box 6) – available to project team members only

- Interactive tools for evaluation such as questionnaires

- Presentations/training materials for the workshops

- Report drafts

- e-mail address for project administration of various

project team members

- “Question-of-the-week” – a non-statistically valid

question on a current topic of interest to all stakeholders;

results available in real time, i.e. as soon as one has

finished “voting”

7. Site Map (Box 7)

- Quick navigation tool for users who don’t want to scroll

through menus

The website will be developed, maintained and hosted at the project head office.

The domain name www.itcevaluation.org will be registered. Website

information will be available in English. The web-site will be built on a module-

based content management system (CMS) – these kinds of solutions are more

flexible and cost effective for updates and new feature modules.

A tree diagram of the web site is shown in the following section overleaf.

12 All the reports produced during the project will be published only after the consent of the

project beneficiary.


Version Final

54

1. About Project 2. Consortium presentation 4. Project documents Contact us

4.1. Evaluation

methodology and

activities

4.2. Different

project reports

4.2.2. Final

Synthesis Report

4.1.1. Interactive

questionnaire

4.1.2. Links

1.4. Links

1.2. Terms of

Reference

1.3. Donors

2.5. Links

2.4. Ticon

Holdings

2.3. ACE Global

2.1. DMI

Associates

2.2. Copenhagen

Development

Consultants

Site map

4.2.1. Desk study

report

4.4. Press

Releases

3. Evaluation team

members

1.1. About ITCQuestion-of-the-week

VOTE!


Version Final

55


Version Final

56

3.9 Quality Assurance

According to the Danida Evaluation Guidelines Chapter 6, at the end of the

project;

the team leader will have to indicate the extent to which:

• Danida guidelines, including the present one, have been used

• the use of methods has been discussed explicitly

• the focus, analysis and results are according to standard

• the language, layout, illustrations, etc are according to

standard

• the professional input of team members is acceptable

A quality assurance plan is attached as an Annex 6.2. The Quality Assurance

Team as proposed in the technical proposal is composed of Denise Colonna

d’Istria from DMI and Annette Munk Sorenson from Copenhagen Development

CA supplemented by the external auditors for validation of paper documents.

3.10 Risks, Mitigating Strategies and

Assumptions

3.10.1 Risks:

The main risks of the project are as follows:

� Lack of access to verifiable indicators in the field related to impact

assessment and result indicators of ITC interventions especially on older

interventions and projects.

� Tight Time frame for carrying out of field studies. The extensive activities

needed for the preparation of the field research as well as the logistics

required to carry out the field research put tremendous pressure on the

resources available during the period.

� Validity of information: there is the risk of collecting biased information

from the stakeholders.

� Reluctance of ITC staff: in cooperating with the evaluators, and

providing the data and information needed for the evaluation

� Difficulties in locating contacts: contacts and individuals who were

involved in completed projects could be impossible to track. Such persons

could provide valuable information, and insight on the impact of projects

in a particular country.


Version Final

57

� Cooperation with Stakeholders: the risk of lack of commitment and

contribution of stakeholders to contribute to the evaluation process

� Variant expectations: Stakeholders to be interviewed may have variant

expectations as to the outcome of the evaluation of ITC, and may be

reluctant to share their view with the Evaluation team

3.10.2 Mitigation strategies

In order to reduce the risks mentioned above, the evaluation team will adopt

the following mitigation strategy:

Participatory process: Stakeholders will be invited to round table/panel

discussions in the evaluation process. This should reduce the risk of the lack

of cooperation and contribution of the stakeholders in the evaluation process,

and create a cooperative atmosphere built on a constructive approach and

openness to the evaluator. This will also provide an opportunity for

validation of facts and discussion of preliminary findings.

Use of control groups: in order to cope with the risk of collecting biased

information, control groups will be selected and included in the evaluation

process.

Use of baseline information and desk study: Maximum use will be made of

other evaluations and published sources of information to establish baseline

information.

Reluctance of ITC staff: Close coordination with SMC has already been

established. The SMC has already issued a memorandum to ITC staff about

the evaluation being carried out and has encouraged staff to provide

requested information to the evaluators. In general the level of co-operation

and support from ITC has been excellent.

Obtaining Verifiable Indicators: Development of detailed Country

Evaluation Tools and Data Sheets and the development of e-questionnaires in

order to generate as much data and information as possible in a coherent

framework, but impact indicators are more difficult to track.


Version Final

58

3.10.3 Assumptions

The following assumptions have been made by the project team:

� Establishing a baseline: the evaluation team assumes that a ten year

historical record for the evaluation process is valid. ITC has

undergone a radical change after a review was carried out in 1994-

1995

� Cooperation and willingness of ITC clients and end users to

participate in the evaluation process.

� Full cooperation from ITC and transparency from management side

� Transparency of beneficiaries during field visit

3.11 Issues/Conclusions

This evaluation project is very ambitious with a global reach. It is important to

adhere to the overall time frame of the project in order that the informal results

can be presented to the informal meeting of the JAG in November. The most

problematic deliverable in terms of the time frame of the study is the completion

of the Field and Technical Study Reports. Preparation of the Field and Technical

Studies Report will require compilation and analysis of the results of Field

Studies by different teams of experts. It will be necessary to verify and cross-

check the results, and this takes time, before they can be considered as validated

findings of Phase II.

The agreed delay in the delivery date for the Field Study, however, will not affect

the timing of other deliverables under the contract. This issue was discussed

extensively and should be treated with a fair amount of flexibility.

An issue related to the budget is that of the costs of the implementation of the

web-site and some of the alternative approaches and tools for evaluation. It was

discussed at the Management Group Meeting in February that the development

of the web site and related communication activities will require additional

funds since these activities were not foreseen in the Terms of Reference.


Version Final

59

4 Project Planning

4.1 Workplan

4.1.1 Methodology, Timeframe and outputs

The following methodology combines the consortium evaluation strategy as

described above with the requirements of the Terms of Reference in terms of

timeframe and output:

I. Inception and Desk Study Phase (January 5- April 15, 2004):

1. Interviews with ITC staff and other organisations

2. Analysis of documentation and reports

3. Preparation of Inception Report

4. Preparation of the Desk Study Report

II. Field Studies (5 months April 15-September 30, 2004):

5. Data collection/stakeholder interviews, including beneficiaries

6. Regional workshops for core evaluators, local consultants and

specialists starting April 13-14, 2004.

7. Production of a briefing note to the annual JAG Meeting, 35th

section, planned for April 2004

8. Production of an initial Report on the Field and Technical Studies,

to include:

• Evaluation of ITC’s intervention in recipient

countries,

• Evaluation of the thematic cross-border programmes

• Evaluation of ITC’s co-operation with other

international organisations

III. Validation Analysis, and Reporting (3 months September 1 –

November 30, 2004):

9. Analysis of the findings presented in the initial comprehensive

report

10. Possible additional fact-finding

11. Production of a Synthesis Report, containing analysis, synthesis

and preliminary conclusions and recommendations.

12. Conduct of a workshop discussing the Synthesis Report

13. Production of notes from the workshop

14. Final conclusions and recommendations elaborated

15. Production of a Draft Final Report.

IV. Finalisation (December 2004 – January 2005)


Version Final

60

16. Reception of comments to draft final report

17. Production of final report

4.1.2 Tasks, Milestones and Reporting

Tasks/Milestones Responsible/Involved Schedule

Documents review TL + Core Team with

iterative consultations

with ST specialists,

management group and

relevant stakeholders

3 months + ongoing

process

Draft Inception report TL 10-02-04

Planning of set-up and

implementation modalities –

team meeting in Geneva

TL + Core team + some

input from ST specialists

From Feb. to March 2004

Planning of field visits and case

studies. Mobilisation of local

consultants

All Key personnel March-May 2004

Team meeting in Geneva All team members 4-5 February 2004

12-20 March 2004

Draft Final Inception Report TL + Core Team + ST

Specialists_

6 April 2004

Desk study report Input of the whole team

Internal QA: TL +

validation with

Management group

7 April 2004

Regional workshops for

preparation of country studies

TL+ Core Team + some

ST specialists

13-14 April 2004

April 21-22, 2004

May 9-10, 2004

June

Field visits implementation:

Pilot visit and full field activities

All key personnel From May to September

2004

Briefing note to Annual JAG

meeting

TL 15.04.04

Preparation of Technical and

Field studies reports

All key personnel August to mid-

September 2004


Version Final

61

Tasks/Milestones Responsible/Involved Schedule

Team Meeting in Geneva

Review of methodologies and

progress of the field and

technical studies.

All team members July 2004

Meetings in Geneva with ITC Team Leader, selected

Team members,

July- September 2004

Submission of Field and

Technical Studies Reports for

validation

Team Leader, Core Team 30 September 2004

Preparation of draft synthesis

Report

TL, Core Team October–mid-November

2004

Presentation and discussion of

field and technical studies with

ITC and with Management

Group and Core Group

TL, Core Team October November 2004

Review of comments TL All team

Submission of draft final

synthesis report

TL, Core Team December 2004

Finalisation of Synthesis Report TL, Core Team January 2005


Version Final

62

5 Recommendations

5.1 Time Frame

The only adjustment to the time frame that is proposed is to delay the

submission date for the Field and Technical Studies Report until September 17,

2004. The other deliverables will be met in the timetable identified in the Terms

of Reference. This has been discussed with the Management/Core Group.

5.2 Budget

We propose a modest addition to the reimbursable expenses budget in order to

pay for the development of the web-site and to implement e-questionnaires and

survey techniques. We propose some reallocation of the budget to accommodate

the translation of some of questionnaires and the materials for field studies. We

also propose some reallocation of the reimbursable funds to provide honoraria

for expert and peer review and to pay for communications costs. Apart from the

website and the administration and tabulation of, e-questionnaires and surveys,

we do not foresee the requirement for additional resources for fees and

reimbursable expenses.

5.3 Communications

In order to provide greater transparency and to promote a broader range of input

and feedback into the Evaluation, we are proposing an active communications

policy including a public website and providing information to the interested

specialized press, to NGOs and to the broader public as well as contact with

Geneva-based delegates, TPOs and other stakeholders in the ITC.

5.4 Mobilisation of Personnel

Within the budget constraints established in the Terms of Reference and the

Contract, the TL will assign missions and tasks to the international evaluators

and the local/regional evaluators for each phase of the evaluation, namely the

desk study phase, the field study phase and the analysis phase. The core team

members and the short term international specialists will be deployed by the TL

as appropriate to undertake specific specialized tasks with specified Terms of

Reference for each phase of the study.

It was proposed that within existing budget restraints one short term

international specialist in Results Based Management and Evaluation should be


Version Final

63

added to the evaluation team to assist in the preparation of the Evaluation

Matrix. Authorization for this additional expert has been granted. Additional

international short term expertise may be required in the course of the

evaluation.

Local and regional experts will be nominated to Danida by the team leader based

on input from the Consortium Members, the team members and other sources.

The local and regional experts will have specific Terms of Reference either

prepared by, or authorized by, the Team Leader.


Version Final

64

6 Annexes

6.1 Annex 1: Evaluation Questions

A ITC’s external environment Desk

research

Research

ITC

Field

Global

Field

Country

1 Global enabling environment

1.1 What are the external factors which may inhibit ITCs ability to achieve its goals X X X X

1.2 What is the development relevance of ITC activities in relation to the overarching

objectives defined in the Doha Development Agenda and the Millennium Declaration?

X X X

1.3 What has been the general performance of multi-agency interventions (IF and JITAP)

within trade and development work? In particular what is ITC's role in these

interventions? How is it evolving?

X X X X

1.4 What was the shift in focus of the WTO, UNCTAD during the past decade? X X X

1.5 How has it influenced the role of ITC? How has it influenced other donor agencies or

international organisations?

X X X

1.6 What is the value-added of ITC’s interventions in relation to the international trade and

development community, particularly in comparison with other international

organizations, such as WTO, UNCTAD and UNIDO?

X X X X

1.7 What is the extent to which it has been possible to achieve concrete results within the

area of trade promotion and the influence of the recent development of the

international trade agenda (trade agreements removal of barriers etc..)

X X X X

1.8 What are the stakeholders views on global policy concerns of trade and development? X X

1.9 What are the constraints due to donor priorities? X X

1.10 How are donor priorities evolving X X

1.11

1.12

How has the Implementation of WTO agreements and market access conditions for

Developing countries developed?

How have broader developments in the trading system such as regional trading

X

X

X


Version Final

65

initiatives affected the role of ITC with respect to the MTS

2 Enabling environment of the host countries ( physical and policy

infrastructure)

2.1 What are the constraints in trade relations in recipient countries? X X X X

2.2 Where have bottlenecks and key potential risks occurred, e.g. in the international,

regional and national enabling environment or in government policies, strategies and

plans as well as national rules and regulations and national institutional capacity?

X X X

2.3 How have national rules and regulations and national institutional capacity affected the

outcome of TRTA interventions and in particular ITC interventions?

X X X

2.4 Does ITC meet the objectives embedded in poverty reduction strategies of recipient

countries?

X X X X

2.5 How has WTO agreement and market access condition influenced developing country

trade policies?

X X X

2.6 What is the capacity of trade support infrastructure - transport, communication and

logistics in LDCs?

X X X

2.7 What is the incentive for developed countries to transfer technical knowledge on Trade

development?

X X

3 Changing needs of the developing countries and countries in transition

3.1 What are the specific changes in needs as perceived by stakeholders in host

countries?

X X X

3.2 What are the needs for diversification of export base – trends in primary commodity

dependence among developing countries, especially least developed ountries?

X

3.3 What are the stakeholders views regarding e-technologies and trade? What is the

preparedness of the country to e-business?

X

3.4 What measures are needed to reduce the digital divide? X X

3.5 How do the needs of countries change according to the level of development with

respect to ITC services?

X X X


Version Final

66

4 Overall availability of Services similar to ITC - TRTA

4.1 What is TRTA? How has is developed in the past and present? X

4.2 What are the subdivisions under trade development where ITC is operating? X

4.3 In what countries and for which income groups? X

4.4 What is the size of the trade development market in volume (number of activities) and

value (USD)? In particular Technical Assistance?

X

4.5 What are the main agencies operating in similar fields? X X X

4.6 How much overlap is there in the target countries/groups? X X

4.7 How to measure TRTA impacts? X X X X

4.8 What are the future trends in TRTA? X X X X

5 Emerging issues

5.1 In any of the above issues, what are the emerging issues relevant to ITC? X X


Version Final

67

B ITC’s intervention for each type of deliverable Desk

research

Research

ITC

Field

Global

Field

Country

1 Specification of portfolio of ITC's deliverables

1.1 For each of the categories listed below, specify the names of deliverables available from

ITC.

1- Tools: include practical guides either generic or local, methodologies and

approaches, benchmarking and assessment tools X X

2- Information sources: include websites, bulletins, databases, online information

systems, e-shop etc..

X X

3- Advisory consists of technical experts who build awareness, conduct a needs

assessment, address issues X X

4- Training: (limited to formal training services and modules provided X X

5- Networks: Defined as visible network, including conferences, exhibitions, e-

networks

X X

6- Programmes, initiatives & projects: (incorporates a number of the above

deliverables, within a time frame and a budget) Programmes can be in preparation

phase, implementation and completed phases?

X X

1.2 What is the relative importance of each deliverable with respect to financial resources

disbursed and target groups over the last 3-5 years?

X X

1.3 How has the product/service been prioritised in view of ITC growth over last 3 years? X X

1.4 For programmes, initiatives and projects, does ITC have an exit strategy for terminating

its activities/programmes in a given country?

X X X

1.5 Has the coverage of deliverables been discontinued? X X X


Version Final

68

2 Outputs

2.1 What are the target groups for the types of products/services offered by ITC? X X X

2.2 How has demand for ITC products and services evolved over the last five years? X X

2.3 How often are the different types of deliverables used? X X

2.4 What are the areas of technical competence where demand has increased for ITC

services over the last 3 years?

X X X

2.5 What is the quality and price of services available? X X X

2.6 What is the outreach of services offered to target groups and countries? X X X X

2.7 How do Global tools adapt to the specific types of developing countries? X X

2.8 What is the uniqueness of the deliverable? X X X

2.9 Is the timeframe for the different interventions appropriate? X X

2.10 Which types of products and services competencies at ITC provide the best synergy

when combined together?

X X

2.11 How important is the element of dissemination of a deliverable? X X

2.12 If a deliverable is fee-based - How much more would the user be willing to pay to use

the deliverable?

X X

2.13 If a deliverable is free - What could be the willingness of users to pay for the

deliverable?

X X

3 ITC target groups

3.1 Who are the main TSIs of ITC?

3.1.1 What is a TSI? What are their main characteristics? (capacity, number of members,

types of services provided, outreach, advocacy)

X X X

3.1.2 What are the new types of TSIs having an increasing role? X X X

3.1.3 How have the skills of TSIs evolved over the last years (Technical skills, capacity to

transfer knowledge, needs assessment/advocacy, project management, promotion, and

networking, e-competence)?

3.1.4 Is there a TSI network: national - international? X X

3.1.5 What is the purpose of the network, how does it operate, who are the members of the

network?

X

3.1.6 What indicators are used to measure the performance of ITC intervention on TSIs? X X X


Version Final

69

How does the TSI access each of the deliverables? X X

What are the strengths/weaknesses/needs of the TSI in its role as an intermediate?

3.2 TSI Relationship with ITC

3.2.1 How close it the relationship of ITC to TSIs? X X X

3.2.2 How much input does the TSI have on development of deliverables? Who identifies

needs? Who identifies sources of finance?

X X

3.2.3 Has the cooperation with TSI ever been discontinued? X X

3.2.4 What is the awareness of TSI of ITCs image, competences and deliverables? X

3.2.5 Who are other providers of support to TSI? X

3.2.6 How important are they compared to ITC?

3.2.7 What are the strengths/weaknesses/needs of ITC activities and deliverables?

3.3 Who are the main end users of ITC?

3.3.1 Define end users: What are their main characteristics (size of business, gender, sector,

in country coverage?)

X X X

3.3.2 What is their influence on need requests? and involvement and contribution to PCM? X X X

3.3.3 How does ITC ensure feedback from end-users? X X X

3.3.4 What indicators are used to measure the performance of ITC interventions at end-user

level?

X X X

3.3.5 How have the skills of end-users evolved over the last years and how has it influenced

the design of ITC product/services (technical skills, capacity to absorb knowledge,

networking, advocacy, benchmarking)?

X X X

3.3.6 How does the end user assess the performance of TSI? X X X

3.3.7 Is the end user aware of ITC? If yes what is the image? X

4 Geographical Focus of ITC

4.1 What are the most important target countries for ITC? X

4.2 What is the geographical coverage of deliverable in question? X X X

4.3 What is the access of ITC to the target groups X X

4.4 What is the outreach of each type of intervention to the direct beneficiaries and end

users?

X X

4.5 What is the experience of ITC in the different countries where it operates? X X

4.6 What are the factors which determine the geographical choice of countries? X X


Version Final

70

4.7 How are priorities made as to the geographical focus of ITC? X X

4.8 Is there continuity in ITC's presence in the countries of operations? how is this ensured? X X

4.9 How are ITC's country activities coordinated? X X

4.10 How are synergies created for ITC activities in a given country? X X

5 Impact of ITC interventions

5.1 How can ITC measure the baseline of a TSI to assess its ability to deliver and ensure

success of ITC intervention?

X X

5.2 What is the actual in-country state of each specific area of competency? X

5.3 What changes has ITC intervention induced on the end users? X

5.4 What changes has ITC intervention induced on the TSI? X

5.5 How has the number of exporters developed during the last 3-5 years? X

5.6 What are the factors which caused this change? What is the contribution of ITC in the

change?

X

5.7 How has the number of sectors in exports evolved over the past 3 years? X

5.8 What has been the export performance (clear evidence of growth, product

diversification, degree of value adding, market diversification) of the sectors of export

during the last 3 years?

X

5.9 What are the factors which influenced this change? What is the contribution of ITC in the

change?

X

5.10 What has been the increase in performance of exporters during the last 3 years

(increase in exports, product diversification, value adding, market diversification)?

X


change?

X

5.12 How has the competitiveness of end-users developed over the last 3-5 years? X

5.13 Do end users belong to advocacy networks? X

5.14 How has advocacy network changed over the years? X


change?

X

5.16 How does the end user/TSI perceive social impact of support services provided? How

are they measured?

X


Version Final

71

6 Performance of ITC deliverables

6.1 Which type of deliverables are most efficient and effective? In quality and quantity? X X

6.2 What are the key factors of success for each type of deliverable? X X

6.3 What are the strengths and weaknesses of each type of deliverable? X X

6.4 Does the channel of delivery affect the performance of ITC outputs? X X

7 Identification and assessment of other Providers

7.1 Who are they?

7.1.1 Which type of deliverables are most efficient and effective for the service provider? In

Quality and quantity? Who supplies them?

X X

7.1.2 Who are the agencies supplying services similar to ITC's? Their modes of delivery?

What are their key areas of competence?

X X X X

7.1.3 To what extent do private sector providers compete with ITC’ss activities? X X X

7.1.4 How do service providers identify needs and prioritize them? X X

7.1.5 How much overlap in activities/target groups is there in the countries? X X X X

7.1.6 How do other agencies measure the impact of their activities? X X X

7.1.7 What are their key areas of focus? X X X

7.1.8 New comers: Are there potential providers of services similar to ITC? In Which areas? X X X X

7.1.9 How do they differentiate themselves from similar providers? X X X X

7.1.10 What types of bottlenecks do similar agencies encounter in the process of PCM? X X

7.2 Degree of competition

7.2.1 In terms of "price"/quality range how are other service providers "competitive"? X X X X

7.2.2 What is the degree of overlap in interventions by service providers? X X X

7.2.3 What is the competitive advantage of ITC compared to other providers of similar

services?

X X X

7.2.4 What are the strengths and weaknesses of other service providers compared to ITC? X X X

7.2.5 What are the key factors of success for the different types of deliverables provided? X X


Version Final

72

C Managerial Issues of ITC (Managerial Challenges) Desk

research

Research

ITC

Field

Global

Field

Country

1 Strategic coherence between interventions, policies and objectives

1.1 How does ITC define its objectives ( mechanisms, WTO, UNCTAD?) X X

1.2 To what extent are ITC policies adapted to the achievement of the objectives? X X X

1.3 How clear, explicit and measurable are ITC objectives? X X

1.4 Is the ITC management performing efficiently in achieving the objectives and reaching

the goals?

X X

1.5 What it the Management strategic planning capacity? X X

1.6 What is the mgmt. degree of strategic awareness and understanding of environment? X X

2 Strategic Planning

2.1 What is the Capacity of ITC to coordinate and make use of potential synergies with

other actors at HQ and Field level?

X X X X

2.2 How does Mgmt. determine the evaluation and control parameters? X X

2.3 Are control and information systems used for performance management and strategy

development?

X X

2.4 Is the response to the external needs built in the organisation of ITC? X X X

2.5 What is the capacity of mgmt. to integrate lessons learnt responding to evolving and

managing growth?

X X

2.6 How is strategy formulated across divisions? Are there performance indicators for each

division?

X X

2.7 How are needs assessment carried out and priorities made in the approval of projects? X X

2.8 What are the key indicators that management should rely on in its decision making?

(Paretto's 80/20)

X X

2.9 How does ITC management ensure quality and efficiency across the organisation? X X

2.10 How are synergies identified and developed at the different levels of ITC? X X


Version Final

73

2.11 Has management identified areas where efficiency can be improved at ITC? in which

areas? What are the measures used to improve efficiency?

3 Promotion capacity

3.1 How does ITC promote itself to the target groups and countries? X X X

3.2 How does ITC promote its activities/competencies to Donors? X X X X

3.3 How is promotion carried out? At HQ level and at the country level? X X X

3.4 What is the experience of ITC with promoting interventions at country levels X X X

4 Networking and partnerships capacity

4.1 ITC Networking with other agencies: who are they and how important are they? X X X X

4.2 How is ITC's performance in cooperating with different partners at HQ level? X X X X

4.3 How is ITC's performance in cooperating with different partners at country level? X X X

4.4 What kind of networks has ITC established at country, regional and international level? X X X

4.5 What is the experience of ITC with promoting interventions at country levels X

4.6 How does Management ensure the anchoring of partnerships at institutional level? X

4.7 How does Management ensure the anchoring of networks and contacts at institutional

level?

X

5 Quality issues

5.1 Does ITC have a policy for Quality assurance? X X

5.2 How is quality ensured at ITC? Who is responsible for Quality issues? X X

5.3 How is the notion of quality ensured at the different levels of the organisation? X X

5.4 How is the quality of inputs and outputs ensured? X X

5.5 How are corrective actions taken in case situations where quality requirements are not

met?

X X


Version Final

74

D Functional issues Desk

research

Research

ITC

Field

Global

Field

Country

1 General questions related to ITC organisation

1.1 How has the org. structure developed since last review? X X

1.2 What are the main features of the present organisation structure? X X

1.3 How are the dept. Involved at each phase of the Project Cycle Management X X

1.4 Are there bottlenecks in the process of PCM? X X X X

2 Operations

2.1 How are the operations of ITC for providing deliverables organised? X X X

2.2 What are the main steps for ITC operations? From project identification to

implementation

X X

2.3 Which sections and department are involved in the process?

2.4 What is the degree of autonomy of the staff in creating and managing projects? X X

2.5 Does the deliverable fully take advantage of ITC's core competences X

2.6 Are there any areas where there is need to improve efficiency of work of ITC

department staff? Which areas?

X

2.7 What types of improvements could be used to improve the efficiency of operations? X

3 Product development

3.1 The ability of ITC to develop new services and products? X X X

3.2 The degree of market research and development for Product dev. Carried out? X X X

3.3 How ideas are generated across functions at ITC? X

3.4 How close is product development to core competences? X X X

3.5 How are new ideas tested and planned? X X

3.6 How are priorities made in view of increasing demand of ITC products/services? X X


Version Final

75

4 Human Resources Management

4.1 How is the HRD organised internally in ITC? X

4.2 How are ITC HQ competences and skill gaps identified and what measures are taken

to fill the gap?

X

4.3 What measures are undertaken by to build up and sustain capacity at ITC HQ? X

4.4 What is the main profile of ITC technical staff: specialists, generalists, multi disciplinary X X

4.5 How is the institutional memory ensured at ITC? X

4.6 What is the age profile of staff at ITC? X

4.7 What is the motivation level of staff at ITC? X

4.8 How does ITC ensure appropriate training of its staff? X

4.9 How is the distribution of "L" and "P" staff at ITC? What is the influence of the

distribution on the activities and operations of ITC?

X

4.10 How are local consultants identified and selected by ITC? X

4.11 Who ensures the quality and operational compliance of local consultants? X

4.12 How is the experience collected by ITC Consultants anchored at ITC? X

5 Financial Resources Management

5.1 Is the level of financing appropriate to the existing needs of ITC? X X

5.2 What is the predictability in funding and flow of funds from ITC pt of view (multi-annual

vs. annual commitments

X X

5.3 Does the flow of funds function properly (timing, individual regulations, core funds vs.

earmarked funds and other

X

5.4 Does the existing level of financing allow ITC to meet the increasing demand? X

5.5 To what extent does ITC funding modality influence the efficiency, quality and quantity

of ITC activities

X

5.6 To what extent does ITC funding modality influence managerial decisions and priorities

in the selection of countries and development of services?

X

5.7 Are the requests for funding timely seen from the donor's perspective? X X

5.8 What are the key Financial constraints which affect the performance of ITC? X X


Version Final

76

5.9 What is the price structure followed for products/services? How is it set? X X

5.10 Do the financial data and reporting system provide information for the assessment of

the financial outcome of individual projects?

X X

6 Monitoring and Evaluation

6.1 What are the management information tools available at ITC? X X

6.2 Is the information system adequate to the needs of the organisation? X X

6.3 Are the monitoring systems sufficiently developed to allow management to take

appropriate actions timely?

X X

6.4 How do internal systems for follow-up, monitoring, evaluation and feedback work? X X

6.5 How adequate are the monitoring systems to the activities of ITC at the different

levels?

X X

6.6 What are the shortcomings of the existing information system? X X

6.7 Are the ITC result indicators relevant and appropriate for the ITC "result based

Management"?

X X

6.8 How are the evaluations of ITC interventions decided? X X

6.9 What are the mechanisms followed for evaluation of ITC interventions? X X

6.10 How does ITC ensure that the results of the evaluations are being used by the

organisation?

X X

7 Procurement

7.1 How is ITC involved in procurement activities? Who is responsible for project

procurement?

X X

7.2 How important is project procurement in individual projects? X X

7.3 Is there an issue between procurement and time frame in PCM? X X

7.4 How does the individual staff carry out project procurement? X X


Version Final

77

E Organisational capability Desk

research

Research

ITC

Field

Global

Field

Country

1 Technical skills and competencies

1.1 What are the key technical competencies needed at ITC? X

1.2 What are the technical competencies available at ITC X

1.3 What are the competencies which are outsourced by ITC X

1.4 How is the match between the competencies needed and the ones available? X

1.5 How is the capacity building of ITC competencies ensured? X

1.6 How is the institutional memory of experience gathered by local consultants ensured? X

2 Project Cycle management - of a particular type of output

2.1 For a sample of output, what are the different stages of PCM? X X

2.2 Who are the parties involved at the different stages of PCM? X X

2.3 What is the typical time frame of Project cycle? X X

2.4 What is the type of information needed at each stage of PCM?

2.5 Is there consistency of projects and programmes with overarching policy objectives X X

2.6 Is PCM demand driven, expert driven, donor driven? X X

2.7 What is the capacity of staff to evaluate and follow-up, and ensure continuity and

feedback (contrasting capacity building as a long term goal, with ITC's short term

framework)?

X X

2.8 To what degree is there coordination with Partners in the use of PCM? X X

2.9 What are the bottlenecks? X X

2.10 Are the objectives for projects and programmes at ITC clear and realistic? X X

2.11 How are the quality factors identified to enhance project benefits and sustainability? X X

2.12 What is the influence of E-technology on the management cycle of project? X X

2.13 what is the quality of information available especially from beneficiaries and target

groups for PCM development?

X X

2.14 How does ITC ensure continuity after termination of a project? X X


Version Final

78

3 Transfer of knowledge and capacity building

3.1 What are the modes of transfer of knowledge and skills? X X X

3.2 How does capacity building tie in with the overall mission of ITC? X X

3.3 How does ITC ensure that transfer of skills is integrated in its programmes and

projects?

X X X

3.4 How is the transfer of skills organised? X X X

3.5 How are needs and demand reflected in the deliverables? X X X

3.6 Who at ITC is responsible for managing and arranging for the transfer of skills? X X

3.7 Who at ITC is responsible for developing the capacity building element in the

deliverables?

X X

3.8 Who at ITC is responsible for ensuring the quality of the transfer of knowledge? X

3.9 How does ITC select and ensure the relevance of the target group (recipients)? X X

3.10 How relevant in terms of capacity building are the deliverables offered by ITC ? X X X

3.11 How is the quality and quantity of capacity building elements in deliverables as

compared to other providers?

X X X

3.12 What kind of tools is ITC using to assess the capacity building? X X

3.13 How is ITC ensuring the quality of the staff involved in knowledge transfer? X

3.14 How is ITC monitoring the training (and other activities) of the intermediates? X

3.15 How is ITC ensuring that developed capacity of the TSIs is sustained? X X

4 E-competence

4.1 How has the e-technology affected the intervention of ITC and provision of services? X X X

4.2 How does e-technology influence the linkage to end-users and impact on the field? X X X

4.3 How efficient is e-technology as a method of training compared to traditional types of

interventions?

X X

4.4 What is the impact of e-competence technology as compared to other types of

interventions?

X X

4.5 How does ITC cope with e-reluctance? X X

4.6 How is e-competence built in the ITC organisation? X X


Version Final

79

F ITC’s comparative advantage Desk

research

Research

ITC

Field

Global

Field

Country

1 Core competencies of ITC

1.1 What are the core competencies of ITC? List them (15 competences of ITC) X X X

1.2 What is the involvement of ITC in technical areas related to development concerns? X X

1.3 How have these competencies evolved over the last years? X X

1.4 What is the relevance of core competencies to existing and future demand? X X X

2 Identification and assessment of other agencies of TA

2.1 Who are the agencies supplying similar services to ITC? In which areas? X X X X

2.2 How do the competencies of ITC compare to other international organizations, eg

UNCTAD and WTO

X X X X

2.2 To what extent does ITC compete with private sector providers? X X X X

2.3 What is the comparative advantage of ITC on other providers of similar services? X X X X

2.4 How is ITC differentiating its products from similar providers? X X X X

2.5 How much overlap is there in the target countries/groups X X X X

2.6 How do other agencies measure the impact of their activities? X X X X

2.7 Are there potential providers of services similar to ITC? X X X X

2.8 In terms of "price"/quality range how are ITC services "competitive"? X X X X

2.9 What types of bottlenecks do similar agencies encounter in the process of PCM? X X X

2.10 Where do other agencies focus their activities?

2.11 How do other agencies identify needs and prioritize them?


Version Final

80

3 Access to Demand

3.1 How does ITC assess needs from target groups? X

3.2 How does ITC respond to external needs? X X X

3.3 How are TSIs selected? X X

3.4 How does ITC ensure the match between needs and service provision? X X

3.5 How is demand monitored by ITC staff ? X X

3.6 How are opportunities identified by ITC? X X

3.7 How are new types of needs identified on a continuous basis? X X

4 Positioning ITC in the development community

4.1 Which type of activities and interventions are most efficient and effective? X X X X

4.2 What is the share of ITC in TRTA in the areas related to its competencies? X X X X

4.3 Where does ITC have a competitive advantage and comparative advantage? X X X X

4.4 Is ITC a "niche" player? X X


Version Final

81


Version Final

82

6.2 Annex 2: INDICATORS MATRIX

TRTA and ITC

RB logic of

intervention

Indicators

(available in

Business Plan

2003-2005)

Suggested revision

( to be improved during field

study phase)

Quantitative

measurement

Qualitative

measurement

ITC target

( when

available)

2003 2005

Ultimate goal

(MDG 8)

Develop a

global

partnership for

sustainable

development

and poverty

alleviation

through an

open-trading

system that is

rule-based and

non

discriminatory

ITC’s statement

1. Awareness of relevance of

trade performance to overall

development

2. Improved employment and

region / poor outreach linked to

enterprise export expansion

3. Countries advocacy

network’s capacity and

effectiveness in building a rule-

based and non discriminatory

MTS

1. In country –

volume of Donor

support (in

financial terms)

per type of

support in TRTA,

per donor (of

which ITC)

2. Total number of

employees in

exporting

enterprise end-

user of ITC’s

services, of which

women and youth

(less then 25 yrs

1. Resource

situation / change

both in public and

in private sector

(chamber of

commerce/

industry

associations )

members of Trade

support network

(growth,

stagnation)

2. List of benefits

perceived by

exporting end-

users in regards to


Version Final

83

for this goal:

Develop a

global

partnership for

sustainable

development

and poverty

alleviation and

help the

developing

countries/transi

tion world

benefit

from a rule

based non

discriminatory

multilateral

trading system

(MTS)"

old); number of

enterprises

located in

decentralised

region of the

country

3. Year of

application to

WTO accession

and actual status

development

concerns and what

is their

measurement unit

(or approximate

quantification

measure they use)

to assess it. List of

type of costs and

incentives for

them.

3. List of topics

brought up by

developing

countries to MTS

(10 years period)


Version Final

84

TRTA and ITC

RB logic of

intervention

Indicators

(available in

Business Plan

2003-2005)

Suggested revision

( to be improved during

field study phase)

Quantitative

measurement

Qualitative measurement ITC target

( when

available)

2003 2005

TRTA corporate

goal

Enhance

enterprise

competitiveness

and national /

sector export

performance

ITC mission

statement:

To support

developing and

transition

economies, and

particularly

their business

sector, in their

efforts to realise

1. Trade performance at

a) country scale (source:

country map)

b) sector scale ( detailed

sector/sub-sector)

c) client scale (enterprise

end-user of TSI

networks) ( source: FIT

tool)

2. Share of ITC in TRTA

at country scale

1. a) Country

trade

performance

index over a 10

years

b) Number of

export sectors

2) Volume and

financial

resources of ITC

compared to

other donors in

TRTA, at

country scale.

1. TSI network’s

assessment of export

performance against 5

criteria:

- Value addition in export

markets (growth,

stagnation)

- Diversification of export

markets (yes / no)

- Diversification of

products (yes / no)

- Growth of export

(growth, stagnation,

reduction)

- Supply response capacity

to potential export growth

(available, not)

a) at country scale


Version Final

85

their full

potential for

developing

exports and

improving

import

operations

b) at sector scale

c) at client (enterprise/end-

user) scale

1. Ranking of trade

performance at country,

sector, client scale, from 1

to 4:

1- 0 or 1 criteria out of 5

are positive

2- 2 criteria out of 5

3- 3 criteria out of 5

4- 4 or 5 criteria our of 5


Version Final

86

TRTA and ITC

RB logic of

intervention

Indicators

(available in

Business Plan

2003-2005)

Suggested revision

( to be improved

during field study

phase)

Quantitative measurement Qualitati

ve

measure

ment

ITC target

( when available)

2003 2005

ITC corporate

goals ( 5)

1. To facilitate

the integration

of enterprises

into the MTS

Establishment of

national core

expertise in MTS

issues:

- good

understanding of

product and

market reality

and potential

- national support

network in

existence and

level of relevant

support

- Business sector

involvement in

the development

of country

positions on MTS

Indicators identified are

mainly related to outputs

directly under the

control of ITC. Impact

indicators related to

these goals are suggested

for Technical assistance

capacity building

Measurements will be

made during evaluation

and compared to past

and future situation as

perceived by the field and

compared to ITC’s target

set based on their

financial capacities.

1. Number of country networks

established or supported under

World Tr@de Net and related ITC

programs

2. Number of network meetings,

information bulletins, responses

and contributions to ITC web

sites

3. Number of institutional

committees established in African

countries participating in the

JITAP.

1. 50

2. 60

3. 8

1. 70

2. 120

3. 16


Version Final

87

issues

2. To support

national efforts

to design and

implement

trade

development

strategies

Trade

development

strategies that

take into account

supply capacity,

international

demand and

commercial

practices

Established and

implemented

with active

cooperation of

public and private

sectors in national

and sectoral

strategies

1. Number of trade development

strategies through ITC ( national

or sectoral) that are under

implementation

2. Number of countries assisted in

incorporating an e-dimension into

export strategies

3. Number of requests for pilot

project under the Export poverty

led program

1. 17

2. 4

3. 8

1. 57

2. 14

3. 16


Version Final

88

TRTA and ITC

RB logic of

intervention

Indicators (available in

Business Plan 2003-2005)

Suggested

revision

( to be

improved

during field

study phase)

Quantitative measurement Qualitati

ve

measure

ment

ITC target

( when available)

2003 2005

3. To strengthen

key trade

support

services, both

private and

public

Reinforcement of national

trade support institutions

which provide business

development and

competitiveness

enhancement services to

the business community

- scope of trade support

services provided by TSIs,

including gender-related

programmes

- expanded customization

of generic tools under the

ITC product-network

approach and network

activation

1. Number of TSI partnering with

ITC

2. Number of institutions in

number of countries applying ITC

support tools

3. Number of demands for trade

related information

1. 400

2. 20

institutio

ns in 20

countries

3.

820,000

visits in

157 links

1. 600

2. 60

institutions

in 40

countries

3. 900,000

visits and

175 links

4. To improve Improvement of trade 1. Number of enterprises 1. 1250 1. 2,750


Version Final

89

export

performance in

sectors of

critical

importance and

opportunity

performance in selected

product and services

sectors

- business opportunity

developed as a result of

supply-demand

matchmaking and other

support activities

- contribution of ITC

research and product or

market studies to sector

strategies, TSIs, SMEs

participating in ITC buyer – seller

meetings and matchmaking

activities

2. Number of countries with

partners using ITC strategic

market analysis tools

3. Number of trade in services

associations established

4. Number of institutions

applying ITC developed

management, export and e-audit

tools

2. 128

3. 2

4. 4

2. 140

3. 7

4. 34

TRTA and ITC

RB logic of

intervention

Indicators (available in

Business Plan 2003-2005)

Revision

during field

study phase)

Quantitative

measurement

Qualitative

measureme

nt

ITC target

( when available)

2003 2005

5. To foster

international

competitiveness

within the

business

community as a

Increase in

entrepreneurship skills for

exports and in

competitiveness at the

enterprise level

- level of demand among

1. Number of partners

institutions delivering

business management

training programs

2. Number of enterprises

1. 45

2. 100

enterpris

es in 70

countries

1. 90

2. 120 enterprises


Version Final

90

whole and the

small and

medium-sized

enterprise (

SME) sector in

particular

export managers for

training, skill development

and advisory services from

local institutions

partnering with ITC

- extent of specialized

competitiveness support

programmes available to

SMEs through local

organisations and

networks supported by

ITC.

associations in number of

countries applying ITC

developed support tools

3. Number of trainers

trained and regional hubs

of networks launched

3. 300

trainers

trained

and 2

regional

hubs or

networks

launched

associations in 90

countries

3. 1000 trainers

trained and 6

regional hubs or

networks

launched


Version Final

91


Version Final

92

TRTA and

ITC

RB logic

of

interventi

on

Indicators

(available

in Business

Plan 2003-

2005)

Suggested

revision

( to be improved

during field study

phase)

Quantitative

measurement


( when

available)

2003 2005

TA / CB

specific

objective:

Capacity

building

of

Intermedi

aries (TSI)

and end-

users

in one or

the other

of 15

areas of

technical

1. Multiplier effect

of intermediaries

on the number of

exporting

enterprises,

particularly SME

and women

2. Changes in trade

support services

profile at national

network scale

3. TSI perception

on changes in their

capacity in

respect to an ideal

1. Number of

sustainable

exporting

business

opportunities

developed

through TSI

support, with an

improved

performance of

the enterprise,

per type of

enterprise (big

or SME) and of

which owned by

women and per

sector of export,

2. List of network activities and stakeholders

ranking of network relevance and

effectiveness in carrying each of them (

current state)

1. Minimal achievement of the

objectives

2. Limited achievement of the

objectives, with limited

understanding of needs

3. Good progress towards

achievement of objectives, and

capacity to move on, with support

4. Thorough understanding of needs,

network demonstrate originality and

insight and produce work of good

quality.


Version Final

93

competenc

es in trade

and 5

areas of

technical

competenc

e linked to

developm

ent

concerns

profile identified

as follow:

- Technical skills

(one of the 15 +5

areas of

competence)

- Capacity to

transfer

knowledge

- Needs

assessments /

advocacy and

diagnostic

- Project

management,

internal

coordination and

follow-up

- Promotion to the

client and to the

donor

- Networking

per TSI

2. Number of

TSI members of

the network

(and

characteristics)

(current, last 5

years and

foreseen in the

next 5 years)

2. Type of

support services

provided and

number of users

per type of

services

2. TSI network ranking with respects to

criteria:

- multilevel trade network (geographic and

sector)

- regular meetings

- recognition and influence with policy

makers

focus covers all competitiveness issues (not

only volume of export)

- management of TSI: leadership,

communication and consultation, capacity to

mobilize additional resources or generate

revenues, M&E mechanisms, first stop – not

one stop.

- International leveraging at regional and

global scale

3. TSI capacities ranking (1 to 4), compared

to 5 years ago and foreseen in 5 years.

(sampling of individual entities depending

on the number of TSI in the country) 1. not

used 2. used with difficulty 3. good

capacity 4. leader in the country


Version Final

94

TRTA and

ITC

RB logic

of

interventi

on

Indicators

(available

in Business

Plan 2003-

2005)

Suggested

revision

( to be improved

during field study

phase)

Quantitativ

e

measureme

nt


( when

available)

2003 2005

TA / CB

specific

objective:

(continue)

Capacity

building

of

Intermedi

aries (TSI)

and end-

users

4. Current state in

country of

technical

competence with

respect to future

profile foreseen

4. TSI ranking of current actual state in country ( 1 to

4,same criteria)

1. Export strategy: available in country, under

implementation and involving business sector in the

development of country position on MTS

2. MTS: capacity of the TSI to understand the

business implications of the trading system,

negotiate and draft contracts, resort to arbitration /

mediation services

3. Greater capability of enterprises to comply with

market requirements, seek contracts, negotiate and

exploit opportunities, assess its competitive position

and trade potential

4. Full potential of export

5.Market analysis and strategic market research

capacities within the network

6.Public – private partnership and networking


Version Final

95

growing

7.Trade in services


9. E-competence

10. Ensure and manage trade finance, access of SME

to trade finance

11. Quality products and meeting standards of

international markets

12. Supply chain management and import

13. Export packaging: label

14. Legal aspects

15. Institutional strengthening of TSI: management

capacities of TSI

16…

a) environmental sustainability

b) women in development

c) poverty eradication

d) economic and technical cooperation among

developing countries

e) digital divide


Version Final

96

TRTA and ITC

RB logic of

intervention

Indicators

(available

in

Business

Plan 2003-

2005)

Suggested

revision

( to be

improved

during field

study phase)

Quantitative

measurement


( when

available)

2003 2005

Technical

assistance

outputs

6 types of

Deliverables:

• Tools

• Information

sources

• Advisory

• Training

• Network

• Programs,

projects,

initiatives

1. ITC’s

portfolio of

products

2. Uniqueness

of ITC

deliverables

compared to

its competitors

3. Adequacy

1. Percentage of

disbursement and

number of projects at

country level per area

of competence, channel

of delivery, donor, and

compared to requests

3. Number of requests

in ITC per products

5. Percentage of

disbursement of their

TRTA portfolio and

number of projects

5. Numbers of TSI

1. List of competences delivered in the

country / list available at ITC

1. Quality ranking (1 to 4) by the TSIs of

each delivered product in respect to a

previously established of quality criteria of

what would be a good product from the

perspective of the stakeholder. Product

quality ranking:

1. failure, achieve very few number

of criteria

2. achieved half to most of criteria

3. Achieved all criteria

4. achieved more than all stated

criteria

2. List of ITC delivered products and


Version Final

97

with TSI needs

4. Adequacy

with End-user

needs

reached ( and

characteristics);

number of end-users

reached (and

characteristics)

uniqueness factor identified by TSI,

compared to list of competitors most

important deliverables and uniqueness of

their product

3. List of TSIs’ demand (what they asked

for)

3. Open list of area of competence available

and TSIs demand ranking (1 to 4) per area

(criteria for demand ranking to be finalised with

stakeholders)

3. List of type of deliverables, and TSIs

demand ranking

3. List of problems faced in TSIs business

4. List of problems faced by exporting end-

users and by potential exporting end-users

5. List of area of competence / list available

in competitors

5. List of types of deliverables / list of

available in competitors


Version Final

98

TRTA and ITC

RB logic of

intervention

Indicators

(available

in Business

Plan 2003-

2005)

Suggested

revision

( to be

improved

during field

study phase)

Quantitative

measurement


( when

available)

2003 2005

Target groups

- Trade support

institutions (TSI):

network and

individual entity,

also referred to as

intermediaries in

reaching the

enterprises end-users

of ITC services.

- end-users:

enterprises,

particularly Small

and Medium size

(SME)

See targets

indicated at

the level of

goals

1. Access of

ITC to

intermediaries

2. Visibility of

ITC and access

to deliverables

(see master

document)

3. Outreach

4. Cost and

sustainability(

willingness of

intermediaries

to pay for a

1. at country

scale: Number

of TSI contacts

in the country

and % per types

of TSI

1.at ITC

portfolio scale:

number of

countries

reached by

channel of

delivery and

main

deliverables in

each channel

1. List of TSI contacts of ITC with their

characteristics

• Private or public,

• Which sector,

• Which area of competence,

• Centralised, decentralised or both.

• Number of years in TRTA business

2. List of deliverables of ITC in country and

number of TSI met per level of knowledge

(awareness, know-how, practice)

2. Number of TSI met per means of access

used (internet, partner contact, ITC direct

contact, TSI network contact)

4. Willingness ranking of TSI to pay for ITC


Version Final

99

TSI can be:

Partners institutions

Enterprises

associations

Trainers

Enterprises

Individuals visiting

the e-sites

Countries

deliverable)

3. Number of

end-user direct

beneficiaries of

TSI using ITC

products

services ( ranking 1 to 4 to be determine

with stakeholder in the field)

4. Willingness ranking of end-users to pay

for support service


Version Final

100

TRTA and

ITC

RB logic of

intervention

Indicators

(available

in Business

Plan 2003-

2005)

Suggested revision

( to be improved during

field study phase)

Quantitative

measurement

Qualitative

measurement

ITC target

( when available)

2003 2005

Inputs:

Financial

resources

1. Funding modalities

2. Cash-flow

considerations

3. Beneficiary cost-

sharing and willingness

to pay

4. Pricing of ITC’S

products

5. Cost-effectiveness

1. % of budget per

source of funds

2. Time frame of

financial resources

4. % of saleable services

of total budget

5. % of disbursements

per deliverable, per

channel of delivery

(allocation of financial

resource) ,

5. overhead / operation

ratio

Human Strengths and


Version Final

101

resources weaknesses of ITC’s

human resources

Management

1. Project cycle

management of outputs

result-based oriented,

demand driven

2. Strategic coherence

between interventions,

policies and objectives

3. Promotion capacity

4. Networking and

partnership capacity

5. Quality, monitoring

and control capacity

1. Time frame of

delivery of each phase

of the project cycle

management (PCM) of

output

1. List of phases of

PCM of outputs for a

sample of main

deliverables

2. Service responsible at

each phase of the PCM

of output and human

resource involved

3. Information use at

each phase

4. Level of coordination

with partners

5. Level of coordination

within ITC

6. Strengths and

weaknesses of the

process in respect to

relevance, efficiency,

effectiveness and

sustainability


Version Final

102


Version Final

103

Selection of Countries and Projects for Evaluation

The following section has elicited much discussion, but the original proposal has been

retained to describe the evolution of the approach to sampling.

The scope of this Joint Evaluation includes field- assessments of ITC’s activities

and results in selected countries. The presence of ITC interventions in nearly 170

countries, and the large range of ITC products and services that are used by ITC

clients, call for a process of country selection that is highly representative and

balanced in terms of the following criteria:

• Geographical distribution of ITC actions (representing the proportion of

delivery, as stated in ITC annual reports)

• Type of products and services supplied by ITC (selection covering the

flagship products of various ITC divisions, including global and country-

specific services)

• Economic development status of beneficiary countries (a balanced mix of

LDCs as well as OLICs, also large and small countries)

• Coverage of all ITC programmes, including the programmes supporting

the UN common areas of concern, and the multi agency programmes

such as JITAP and IF

• Beneficiary countries under different income profiles

• Trade performance (countries that have recorded consistent, high growth

in GDP and in trade in absolute terms, as well as countries that have not)

The rather limited budgetary and time resources for this Evaluation place

constraints on the number of countries that can be taken up for comprehensive

field evaluations. At the same time, it is expected that the countries chosen for

field evaluations cover all the above groupings, to provide deeper insights into

the results and impact of the interventions, and the challenges that ITC might

face in effective delivery of its services, on account of external factors.

The country level evaluations are expected to provide useful field information to

support answers to the following questions:

a. How far has it been possible to achieve concrete results- outputs and

outcomes-within the area of trade promotion?

• What has been the outcome, efficiency and relevance of interventions

at national, regional and international level targeted towards

i. Trade policies and strategies

ii. National infrastructure for trade promotion, import

management

iii. Support services for trade promotion


Version Final

104

iv. Training institutions

v. Sector and enterprise level

b. Where have bottlenecks and key risks occurred?

i. In the international, regional and national enabling environment or in

government policies, strategies and plans as well as national rules and

regulations and national institutional capacity?

ii. How have they affected the output and outcome of ITC activities?

c. Which type of activities and interventions are most efficient and effective-

i. Horizontal/multilateral efforts or targeted interventions in individual

countries?

ii. Quality versus quantity

iii. Does success of an intervention depend on the region and/or the level

of development?

iv. Is the time frame for different interventions appropriate?

To capture insights and to answer some of these aspects, it is also desirable to

compare experiences and results in countries that have received the same or

similar interventions from ITC.

6.2.1 Project Universe for Evaluation

The project portfolio is discussed in more detail in the Desk Study.

The ITC Projects database (intranet) documents only those projects that were

operational, closed operationally, or closed financially during 2001-2004. In this

database, there are, in all, 539 projects in the period of 2001-04. Year-wise

information is available for years 2003, 2004 and the aggregate for 2001-2004.The

database provides details for these years, classified by beneficiary country,

responsible division/section, donors, and other useful categories. Because

information is not available for projects before 2001, only projects appearing in

the data for 2001-2004 shall be sampled for the purpose of project selection.

Status of Projects

2001-04 2004 2003 2002 2001

All projects 539 231 265 na na

Operational 300 182 167 na na

Closed operationally 131 36 64 na na

Closed financially 37 13 34 na na


Version Final

105

There is a conflict between the selection of projects based on the objectives of

measuring impact (impact manifests only in the longer-term, after an

intervention has been completed), and the importance of feedback and

interaction with project management units and other stakeholders, which will

have moved on soon after the intervention. It becomes difficult to track and

obtain feedback from stakeholder bodies that participated in an intervention

more than three or four years ago. Therefore, to obtain meaningful results from

the evaluation, equal weight would be given to completed and operational

projects, selected from the 2001-04 database.

Average number of projects per country:

On an average, each country has 3 projects from the project universe for 2001-

2004. This would imply coverage of approximately 27 countries, if projects are

selected on a random basis. But, there is a wide dispersion in intensity across

various countries, ranging from 1 project each in 19 countries to 52 projects in

Tanzania alone.

Of the 170 countries in the project universe, 42 countries have 3 or fewer projects,

aggregating to 74 projects, averaging 1.76 projects each. On the other hand, more

than ten countries in Africa have more than 15 projects each going on since 2001.

Sample size:

To adequately cover all types of services and all profiles of countries, we propose

to select around 15 countries and 80 projects from the database, representing

almost 15% of the relevant project universe.

Choice of Sampling Technique:

Given that the Joint Evaluation’s task is to bring an analysis of the effectiveness

and impact of ITC activities. Given the wide variance in ground conditions in the

countries in which ITC operates, and given the technological difficulties in

building indicators for measuring impact of a specific organization/ intervention

in a development environment having several actors and different types of

interventions, the analysis shall depend more on qualitative and experiential

indicators than quantitative interpretations.

Classical Random Sampling by countries:

The random sampling approach by countries–where each member of the

universe has an equal probability of being selected-is not desirable for the

selection of countries for this evaluation. This selection ignores the fundamental

premise that ITC has a prioritization of countries/ regions for its interventions,

and this prioritization needs to be reflected in the samples. Also, because the


Version Final

106

evaluation proposes to gauge the impact of ITC on trade development, there is a

stronger case for taking up countries where there has been considerable ITC

activity represented by a large number of projects, preferably covering several

services and programmes of ITC. A random sample can reduce the proportion of

such countries, given the wide dispersion in number of projects among the

country universe in the database.

Sampling by projects:

On an average, each country has 3 projects from the project universe for 2001-

2004. This would imply coverage of approximately 27 countries, if random

selection were done by projects. Also, there is almost a 25% chance of getting

countries having three of fewer projects. The field studies of countries with such

low volume of ITC involvement (including different types of isolated activities)

would not be useful in determining the impact of ITC, and there is a high chance

that conclusions from such countries would tend to understate the achievements

of ITC. As a result, the inclusion of countries with marginal ITC activity is not

likely to be cost-effective.

Evaluating impacts in countries with greater intensity of ITC activities is likely to

give useful insights into the types of interventions and combinations of

interventions that are likely to demonstrate better results in developing

countries. For these reasons, it is considered important to include in the sample, a

sufficient number of countries where there has been a meaningful and sustained

presence of ITC.

Sampling by service divisions/activities:

ITC delivers its services through specialised divisions, and each project is allotted

a Responsible Section. The allocation of projects by division and sub division,

and a section wise sample, based on a 15% sampling rate from the 539 projects is

as follows:

Division/Department Staff

strength

Number of

projects

Share % Sample size

based on 15%

selection

OED 18 44 8.1% 6 projects

DTCC/OD ? 6 1

DTCC/OD/LDCU 4 4 1

DTCC/OD/SSTU 11 12 2

DTCC/OA 20 170 26

DTCC/OASEC 8 41 6


Version Final

107

DTCC/OAPLAC 11 32 4

DTCC subtotal 54 265 49.2% 40 projects

DPMD ? 1

DPMD/TSU 6 14 2

DPMD/TIS 17 18 3

DPMD/MAS 35 18 3

DPMD/MDS 17 52 7

DPMD sub total 75 103 19.1% 15 projects

DTSS/OD ? 10 1

DTSS/FASS ? 8 1

DTSS/EMDS 15 20 3

DTSS/IPMS 15 47 7

DTSS/BAS 30 42 7

DTSS subtotal 60? 127 23.5% 19 projects

Total 207? 539 100% 80

A sampling based on the service divisions addresses the geographical dispersion

of activities, but differently than the value distribution of ITC services. According

to the above sampling, there should be 26 projects in Africa, which is a

proportionally higher weight than Africa’s share in terms of value of ITC

projects, which is around 40%; and 10 countries together in the Arab, East

Europe, CIS and Asia Pacific, which is proportionally lower than their value

share (45%).

There is another dimension to the sampling of projects based on responsible

sections, given that two of the divisions- DTSS and DPMD- are technical

divisions, which contribute to service delivery in projects that are undertaken in

the leadership of DTCC. A sizeable share of resources and activities in these

divisions goes into developing and delivering a common pool of products, which

are used to different degrees in various country programmes.

Stratified or Quota Sampling:

Given that the classical random sampling approach will give unrepresentative

samples for this evaluation, we wish to base the selection on a stratified sampling

or quota sampling method, to include at least some countries from each class

appearing under the following criteria:

• Regional: representing the five delivery regions (Africa, AP, LAC, Arab,

EECIS)

• Economic classification: LDC, OLIC, and LMICs as per the OECD

classification for 2001 and 2002 flows (LDC- group of 49 countries


Version Final

108

established by the UN; OLIC - non LDCs with GNP below US$ 760 per

capita in 1998, and LMICs between US $760 and $3030 in 1998)

• Type of economy: market economies and countries in transition

• Population size: small (<10 million), medium/ large (> 10 million)

• Economic performance: countries that posted annual GDP growth and

annual export growth both more than 4.5% during past decade (1990-

2000, appearing in World Bank’s World Development Indicators 2003),

and countries that did not

Because ITC reports its delivery by geographical region, and because each region

has a mix of LDCs OLICs, a quota sampling by regions would give a mix of LDC

and non-LDC countries in each region.

The other classifications for selecting countries are intended to establish control

groups to measure impacts in countries based on different absorption capacities

(size of economy, % of poor population) and based on actual improvements in

trade performance in the past few years.

Another important aspect to be studied is the difference between results and

impacts created by multiple agency/regional interventions, and by single-country

interventions.

Country sample by share of delivery

Region Share in 2001

delivery %

Share in 2002

delivery %

Average share

in delivery %

Number of

samples

Africa 43 43 43 6

Asia Pacific 13 15 14 3

LAC 13 13 13 2

CEE and CIS 19 14 16.5 3

Arab States 5 7 6 1

Total 93* 92 92.5 15

* Global services account for 7% and 8% in these years

Source: ITC annual report 2002

Within each region, the countries shall be chosen on a random basis (picking

chits bearing the country name from a box). Countries that have recent history of

civil disturbances or political turmoil shall be excluded and in case such

countries feature in the sample, there shall be redraws.

At the end of the sampling, countries shall be grouped on the following table to

study the presence of all categories of interest. In case any categories are missing,


Version Final

109

a replacement draw shall be made by deselecting one of the over represented

groups, and selecting until there is representation in all categories of interest. On

12th March, the evaluation team picked a random sample of countries based on

the above method (room 123, ITC office), and came up with the following list:

Results of first stratified random sample

Region Type of economy Size Trade performance

LDC OLIC Transiti

on

Small

M/L Performer Non

performer

Africa

Cape Verde 8 * * *

Mali 20 * *

Ethiopia 19 * *

Namibia 15 * * *

Seychelles 6 * * *

Mozambique 19 *

Asia Pacific

Malaysia 12 * *

Mongolia 6 * *(gdp

only)

Vanuatu 5 * * - -

LAC

Peru 10 * * *

Trinidad &

Tobago 8

* * *

CEE-CIS 3

Turkmenistan 6 * - -

Czech Rep 2 * * - -

Macedonia 1 * * -

Arab

Tunisia 20 * * *

Total at least 2 in

each box

The sample consists of several small island countries, and six of the 15 countries

have less than 5 ITC projects during 2001-2004.

Total database entries: 147

Transition countries: 2

Very small countries: 6

Countries with incomplete data: 4


Version Final

110

Average number of projects: 9, five countries with less than 7 project activity

entries in ITC database for 2001-04.

Therefore, the above sample cannot be considered cost-effective for purposes of

this evaluation. In order to have a more meaningful sample, the stratified sample

for different regions of the world allocated according to ITC budget allocations in

2001 and 2002. The judgemental samples was developed on the basis of meeting

the following criteria:

• Comprehensive coverage of ITC programs and competencies;

• Inclusion of economies at different levels of development; LDCS, OLICs,

and LMICs;

• Coverage of large and small economies; and

• ITC projects that are completed and those that are under implementation.

The following criteria were used to exclude some countries:

• Countries with serious civil disturbances; and

• Countries that had been subject to recent ITC country evaluations.

The result yielded the following list:

Revised list after reiterations:

Country, no of

database records

Income profile Population size Economic Performance

Description LDC OLIC Small

<12m

Med/lar

ge >12m

Performer Non

performer

Africa

Benin, 39 * * *

Burkina Faso, 36 * * *

Kenya, 45 * * *

South Africa, 21 * * *

Tanzania, 52 * * *

Zambia, 22 * * *

Asia Pacific

Viet Nam, 18 * * *

Cambodia, 24 * * *

India, 21 * * *

LAC

Jamaica, 12 * * *

Honduras, 12 * * *

CEE-CIS


Version Final

111

Moldova, 9 * * -

Kyrgyz, 19 * * -

Latvia, 7 * * *

Arab

Tunisia, 20 * * *

Total, 359 5 10 8 7 7 6

The relevant economic data used for the classification appears as an Annex to the

Desk Study.

Sample country groupings by selection profile

Region Income profile Population Economic performance

LDC OLIC Small Medium

/large

Performer Non performer

Africa 6 4 2 3 3 4 2

Asia Pacific 3 1 2 1 2 1 2

LAC 2 0 2 2 0 1 1

CEE-CIS 3 0 3 3 0 1 -

Arab 1 0 1 0 1 0 1

Total 5 10 8 7 7 6

Country fit with major ITC programmes:

JITAP: Benin, Burkina Faso, Kenya, Tanzania,

IF: Cambodia, Tanzania

Sector coverage:

Commodities – coffee, spices, Floriculture – cut flowers, Textiles and Clothing,

Leather

However, there does not seem to be adequate representation of LDC exporters of

textiles and clothing. Textiles and clothing will be addressed in the horizontal

cluster analysis of ITC products and services.

In each of the selected countries, the total number of ITC projects shall be

inventoried and project documents shall be perused and grouped into integrated

projects (IF/JITA) and country specific projects. Within country projects, the

classification of projects shall be made according to the standard ITC

Programmes (7 +3) under which these projects are serviced by ITC.

The five largest projects by value spent in the country shall be taken up for

detailed field evaluations, to demonstrate the possibility of the highest impact in


Version Final

112

each country. Among these, projects that have recently been completed or those

nearing completion shall be given more priority for study, based on the

assumption that it would be easier to locate and meet with the project managers,

principal beneficiaries and other stakeholders in these projects.

Project documents, contact details of key persons at ITC and in the beneficiary

country dealing with the project, and monitoring and evaluation documents if

any, would need to be collected at studied prior to the field mission.

Horizontal Studies:

The countries selected have adequate coverage of all seven ITC Programmes

besides the programmes designed to address the UN common areas of concern.

However, besides the country studies, the evaluation also propose to undertake

field studies at a horizontal level for the major ITC programmes. For some

programmes, the selected countries do not provide a sufficient base to establish

meaningful studies. Therefore, studies on specific programmes shall be carried

out in a few representative countries that are not in the sample.

The following countries have been selected based on the desk study and

successful examples cited by ITC:

• Export led Poverty Reduction Programme: China, Floriculture Project,

Yunan Province; and India, Organic Spices Programme

• World Trade Net: Brazil (Business for Development and UNCTAD XI,

June 2004)

• South- South Trade Programme: South Africa, Kenya

• Export Packaging, Latvia


Version Final

113

Economic data for sample countries to classify size, income and trade

performance

Country Population

2001

thousands

GDP

millions

of

dollars

GDP per

capita

dollars

Annual

growth

total real

product

1990-2001

Export

value

growth

%

1990-2000

Benin 6387 2372 371 4,8 3,4

Burkina Faso 12259 2486 203 4,5 11,9

Kenya 31065 11396 367 2,0 6,3

South Africa 44416 113276 2550* 2,1 2,5

Tanzania 35565 9341 263 3,2 7,4

Zambia 10570 3639 344 0,8 -2,0

Viet Nam 79197 32723 413 7,7 22,7

Cambodia 13478 3404 253 5,0 25,3

India 1033395 477342 462 5,9 9,5

Jamaica 2603 7784 2990 0,2 2,2

Honduras 6619 6386 965 3,1 7,1

Moldova 4276 1479 346 -8,4 -

Kyrgyz 4995 1525 305 -2,9 -

Latvia 2351 7549 3211 -2,2 7,4

Tunisia 9624 19990 2077 4,7 6,0

World avge 6148061 31079820 5058 6,7

Developing countries

and territories

4944824 6526752 1324 4,7 9,1

Countries in Central

and Eastern Europe

336558 806241 2396 -3,9 8,2

Source: UNCTAD Handbook of Statistics 2003

6.3 Note on Country Selection

This section presents the final country selection based on the discussion with,

and comments of, the members of the core group and the management group. In

the Draft Final Inception Report we identified the following criteria for country

selection for country level studies. Thus it modifies the previous section.

First the overall sample was stratified by the relative budget weights of ITC

projects in different regions of the world. Thus, the share of the region in the ITC

budget was used as a guideline for the allocation of country studies among

different regions of the world. Note that this regional distribution ensures

inclusion of economies in transition.


Version Final

114

Distribution of Activities 2001 2002

Africa - 43% Africa - 43%

Asia/Pacific - 13%

Asia/Pacific - 15%


13%


13%

CEE and CIS - 19%

CEE and CIS - 14%

Arab States - 5% Arab States - 7%

Global - 7% Global - 8%

Then within this stratification, a judgmental sample was developed on the basis

of meeting the following criteria:

• Comprehensive coverage of ITC programs and competencies;

• Inclusion of economies at different levels of development; LDCS, OLICs,

and LMICs;

• Coverage of large and small economies; and

• ITC projects that are completed and those that are under implementation.

The following criteria were used to exclude some countries:

• Countries with serious civil disturbances; and

• Countries that had been subject to recent ITC country evaluations

On the basis of these criteria we proposed a sample of 10 countries consisting of

the following countries:


Version Final

115

Original Sample

Country, no of

database records

Income profile Population size Economic Performance

Description LDC OLIC Small

<12m

Med/lar

ge >12m

Performer Non

performer

Africa

Burkina Faso, 36 * * *

Kenya, 45 * * *

South Africa, 21 * * *

Tanzania, 52 * * *

Asia Pacific

Viet Nam, 18 * * *

India, 21 * * *

LAC

St Lucia * * *

El Salvador, 12 * * *

CEE-CIS

Moldova, 9 * * -

Kyrgyz, 19 * * -


Version Final

116

At the meeting of the Management Group and the Core Group in April there was

extensive discussion of the country sample. There was general agreement that

the sample should be reduced. One suggestion was to reduce the number of

country studies to about four as in other joint evaluations.

In our view, -- and it seemed to be the view of some members of the

Management and Core Group --, a larger number of country studies than four is

warranted because of the different types of countries where the ITC is working.

The different types of economies include LDCs, OLICs, LMICs and economies in

transition. The sample also includes landlocked countries and small island

economies.

Perhaps the most important factor suggesting a larger sample is that because the

ITC interventions are dispersed over many economies, the ITC interventions

tend to be small even in countries where they are quite active. While many ITC

competences are delivered in a number of the economies where ITC is active,

some are delivered in fewer economies. The sample of 10 countries provides

broad coverage of ITC competences. Thus, within the stratified sample structure

according to different regions of the world, we have reduced the total number of

countries to ten from fifteen, but we have not reduced it further.

In so doing we have reduced the number of LMICs from 3 to 2, but we still

propose to be evaluating some horizontal project activity in LMICs. The key

criteria for selecting the following sample of 11 countries was the coverage of ITC

competences, delivery tracks and types of deliverables. Please note that the

specification of key competences, delivery tracks and deliverables only captures

the most import project activities. The purpose of this survey of key

competences, key delivery tracks and deliverables is to ensure that the sample is

comprehensively covering ITC interventions. It is evident that the sample is

heavily weighted to countries where the ITC is especially active.

The proposed sample is presented and the coverage of ITC competences for these

countries is summarised in the following table.


Version Final

117

Country Selection Survey Of ITC Activities

Country Income profile Population size

Main areas of

competence

Main types of

deliverable

Name of

ITC outputs Country portfolio

LDC OLIC LMIC

Small <

12m

Med/large >

12 m

Country

portfolio

Country

portfolio

sample of

projects

number

of entry*

number

sampled

AFRICA

1 Benin 1 1 2,4,5,10,11,1 1,2 39

2 Burkina Faso 1 1 2,3,4,5,8,1 36

3 Kenya 1 1 1,2,4,5,7 1,6,3,4

E-trade

bridge,

SSTP,

EPRP,

JITAP,

PAckIT,

Executive

forum 28/45 17

4 Tanzania 1 1 1,7,2,3,4,5,15 6,2,4

IF, 11, 1, E-

trade

bridge,

SSTP, 4, 8,

Jitap,15, 34/52 17

ARAB STATES

5 Tunisia 1 1 2,1,3,4,14 6,4 18/21

ASIA PACIFIC

6 India 1 1 1,9,3,15 1,4,5,6

E-trade

bridge 19/21 9

7 Vietnam 1 1 1,9,2,3 6,2,3 18/16


Version Final

118

Country Income profile Population size

Main areas of

competence

Main types of

deliverable

LDC OLIC LMIC

Small <

12m

Med/large >

12 m

Country

portfolio

Country

portfolio

sample of

projects

number

of entry*

number

sampled

8 Kyrgyz 1 1 1,3,15,7 6,2 19

9 Moldova 1 1 1,7,12,15 4,6 9

LAC

10 El Savador 1 1,2,3,4 1,4,5,6

WTN, E-

trade

Bridge,

comp tool

fair, EPRP 12

11 Saint Lucia 1 1,7,15 5, 6

Executive

strategy 8

TOTAL 3 7 1


Version Final

119

Horizontal &

functional studies

Brazil, Multilateral Trade system and business advocacy 1

Executive

forum, SME

comp -

PROCIP,

PACKit 15

China, Integration of

poverty reduction

TRTA 1 EPRP,

South Africa,

Integration of

development

concerns in TRTA 1 1 C,D,1,3,7,5

E-trade

bridge,

SSTP,

EPRP 21 5 or 9

Turkey or

Philippines,

Capacity building

SME

competitiveness

Latvia, export

packaging

When the cell includes a list: it is in decreasing order of importance ( either frequence, or total amount)


Version Final

120

List of technical competences, delivery tracks and deliverables from the TRTA reference framework

on page 29 of the Draft Final Inception Report

List of areas of

technical

competences

In decreasing order of importance, with at least 3

entries related

List of

tracks

1 Export strategy

2 MTS 1 - Global, generic

3 Enterprise management and competitiveness 2 - Multi progr., multi donors

4

Product development by

sector 3- Country / region specific

5 Market analysis and strategic market research

6 Public - private partnership and networking

7 Trade in services

Types of Deliverables

8

Trade information

management 1. Tools

9 E-competence 2

Information

sources

10 Trade finance 3 Advisory

11 Standards and quality 4 Training

12 Supply chain management 5 Networking

13 Export packaging 6 Programmes

14 Legal aspects

15 Institutional strengthening of TSI


Version Final

121


Version Final

122

6.4 Annex 4: QA Plan

6.4.1 Introduction to the Quality Assurance Management

System

An active Quality Assurance (QA) is essential to ensure the smooth

implementation of the project, the achievement of project objectives, and an

effective liaison with the stakeholders and Danida. In order to secure this,

administrative procedures have been set up to ensure:

6) communication and time for discussion and feed-back between evaluation

team members

7) fast accessibility to information and documents collected and developed

during the evaluation

8) dialogue with Danida, the Management Group, Core Group, and other

stakeholders

The QA procedures have been developed on the basis of Copenhagen

Development Consulting’s QA-system; quality requirements of the tender

document; the consortium’s organisation; and our approach to the evaluation.

The approach to the QM system is that it should be as simple as possible and as

detailed and complex as necessary. If it is too detailed and complex the system

will not be followed; if it is too simple it will not provide the necessary assurance

and documentation of quality.

The procedures will be presented to all evaluation team members (in writing

and orally) at the earliest possible. The first internal audit will be carried out as

soon as possible after the inception period to ensure that the QM system is

understood and being followed.

6.4.2 Administrative procedures

Communication lines and feed-back

Discussions and feed-back between the evaluation team members is found of

crucial importance to the quality of the evaluation. To ensure this, meetings will

be held in the beginning of the evaluation and at important steps in the process.

Fast access to documents and files will be secured through the use of a web-

server with access for evaluation team-members. In addition a proposal for a

public web site which will provide access to key project documents to Danida,


Version Final

123

the Management Group, the Core Group and the Consultative Group, as well as

the general public is included in the Inception Report.

All relevant information from Danida, the Management Group, and the Core

Group will be communicated to the evaluation team; and all information

gathered by the evaluation team will be communicated to TL and relevant

evaluation team members.

Handling of Documents (paper version and electronic)

A filing system has been developed. This filing system will be used both for the

filing of printed documents, and for electronic filing (web-server or filing system

at project head office). Please refer to Annex 5.

� A paper version of all documents/reports in their final version will be printed

and filed at the project head office.

� A copy of all outgoing documents will be filed in a printed version.

� All electronic versions of documents will be filed on the web-siteand/or at the

project head office.

� Information in hard copy only received by the project head office will be filed

and a list of relevant documents sent to relevant evaluation team members. A

list of filed hard copy documents of importance will be kept on the web-

server.

� Information in hard copy only received by individual team members will be

filed by the individual members. If found relevant a copy will be provided to

TL and/or other relevant team members. A list of filed hard copy documents

of importance will be sent by the expert and will be kept on the web-server.

Guidelines for use of the htp-server

What to save where and when:

� The folder system/structure is the same as the filing system (repetition

of numbers and text)

� The filing system is self-explanatory

� QA-folders will be established for relevant reports and other QA-

activities.

� It is suggested to establish folders with documents and/or lists of

information and data collected

When to save and how to get information about new "files"/papers

� all relevant data must be saved immediately

� when important files have been added to the web site all relevant

“members” will be notified in an e-mail (attachments are not needed).

Back-up of content on the server


Version Final

124

� daily, weekly, and monthly back-up carried out by Liga Stikute

� tapes/CD’s are kept in a different location from the server

� If the web-server is down (not functioning) all documents of

communication must be sent to TL who will save on the project head

office computer for later saving at the web-server

� A back-up email system and storage is available on the email address,

[email protected].

Definition of responsibilities among team-members

There is a management process established in the Inception Report. Each

member of the evaluation team will receive from the Team Leader terms of

reference and a mission allocation for each phase of the study, namely the Desk

Study phase, February 15-March 31, 2004, the Field Study Phase, April 1- June

30, 2004 and the Synthesis Report (July 1, 2004-September 30, 2004).

Financial Procedures

Accounting

Financial procedures for reimbursable expenses will be followed according to

Danida guidelines and procedures. Each of the experts and/or consortium

members will submit an original of all financial documentation to the office of

DMI Associates, 21 rue Longue, BP 1176, F -69201, Lyon Cedex 1, France by

registered mail. Photocopies of all documentation submitted must be retained

prior to sending them. Such documentation should be submitted monthly and all

documentation must be submitted quarterly.

Time sheets for professional time should be signed by the expert and submitted

by registered mail to the Team Leader, Murray Smith, 147 rue de Lausanne, CH-

1202, Geneva, Switzerland for signature and approval on a monthly basis. The

Team Leader will submit all time sheets received within seven working days

from the beginning of the month / within nine working days to the office of DMI

Associates 21 rue Longue, BP 1176, F-69201, Lyon, Cedex 1, France.

Guidelines for accounting for duty travels

All duty travel must be approved by the Team Leader. Normally authorisation

will be included in the terms of reference for the expert for each phase of the

study, but all travel must be authorised by the Team Leader.


Version Final

125

General Financial Reporting

While each expert and consortium member is responsible for maintaining

complete financial records, all original documentation should be submitted to

the office of DMI Associates, 21 rue Longue, BP 1176, F-69201, Lyon, Cedex 1,

France. All accounts must be submitted within five days of the end of each

quarter.

Guidelines for Hiring Local Consultants

CVs for candidates for local consultants will be submitted to the Team Leader for

approval. Interviews will be held if feasible, and a report should be submitted to

the Team Leader if an interview is held.

QA-organisation

A QA team has been formed with the following thee members:

Mrs. Denise Colonna d’Istria will be responsible for technical QA of

1) the evaluation process and results

2) review of technical reports

Mrs. Annette Munk Sørensen will be responsible for:

1) development and maintenance of the QA-plan

2) monitoring timeliness and adherence to the QA-plan

Daily QA will be the responsibility of the Team-leader, Core-Team members, and

Team of Specialists.

According to the Danida Evaluation Guidelines (February 1999) Chapter 6,

the team leader will have to indicate the extent to which:

• Danida guidelines, including the present one, have been used

• the use of methods have been discussed explicitly

• the focus, analysis and results are according to standard

• the language, layout, illustrations, etc are according to

standard

• the professional input of team members is acceptable

The Team Leader will advise all experts of the requirements of Chapter 6 of the

Danida evaluation guidelines and they will be asked to sign a letter confirming

that they are in compliance with the requirements of Chapter 6 of the Danida

evaluation guidelines.


Version Final

126

All mission reports, country reports and other reports must be submitted to the

Team Leader for approval.

Documentation of QA:

A QA folder will be created for each report as a sub-folder on the web-server

with back-up with the server of the consortium leader DMI Associates

The QA-folder will contain:

1) QA form for reports,

2) related e-mails saved as rtf-files (Rich-Text-Format),

3) related versions of reports with “track-changes”.

Quality Assurance Plan

Generally speaking, QA is provided through continuous review of reports,

technical papers, meetings, e-mail discussions, etc.; such reviews are compared

with the work plan and administrative procedures. In addition, QA also covers

financial aspects, scrutiny of vouchers, and auditing.

As well as measuring the evaluation progress against the more obvious time and

quantity standards, the evaluation will also be measured against quality

standards given in Danida guidelines for evaluation.

Quality Assurance of reports

A traditional QA will be made for all major reports. As most work will be

distributed and quality checked over the internet, no signature will be found on

the QA forms. Instead QA will be documented by filling in the QA-form

electronically and saving related e-mails and documents with “track changes” in

the QA-folder for the relevant reports.


Version Final

127

In addition, quality will be assured through review and feed-back during the

process of writing the reports. This feed-back will be given by the Team-Leader,

core team members, the team of specialists, and local consultants as found

relevant by the Team-Leader.

The QA-form for reports has been developed based on the requirements given in

Danida’s “Evaluation Guidelines”.

Layout for reports

A format for layout of report has been developed and this must be used by all

evaluation team members. Annex 3 (ITC-report.dot).


Version Final

128


Version Final

129

6.4.3 QA Annexes

Annex 1

QA-Plan (QM 1.v2)

Annex 2

QA form for reports (QM 2.1v2)

Annex 3

ITC-Report.dot

Annex 4

Filing system

Annex 5

Htp-server structure


Version Final

130


Version Final

131

6.4.4 QA Annex 1

Quality Assurance Plan

Activity Responsible Participating QA-

responsible

Date Documentation of QA

Daily QA TL + Core Team On-going e-mail, documents, etc.

QA of TOR for local consultants TL Core Team TL On-going e-mail with attached TOR with comments

Accounting TL OV monthly e-mails with comments

QA of inception report & QA-plan TL Core Team/

DCI 2 days before

dead-line

QA form for reports, related e-mails, and

documents with “track changes”.

Verification of declaration letters OV

QA of desk study report TL Core Team / Team of

Specialists / Local

consultants

DCI 3 days before

dead-line



QA Audit AMS Twice during

the project

QA based on the project intranet site.

QA of field study report TL Core Team DCI 3 days before

dead-line



QA of draft final synthesis report TL Core Team Technical: DCI

Layout: AMS

3 days before

dead-line



QA of final synthesis report TL Core Team Technical: DCI

Layout: AMS

3 days before

dead-line



Legend:

TL Team-Leader

QA Quality Assurance

AMS Annette Munk Sørensen

DCI Denise Colonna d’Istria


Version Final

132

OV Olivier Vautherin, external auditor

Paper versions of this document will be updated in one copy only.

Elaborated by: Controlled by: Approved by: Version: Date of approval:

AMS 1,2 09/02/04

MGS 2,3 10/02/04


Version Final

133

6.4.5 QA Annex 2 QA of Reports

Report:

Author(s):

[name + date and subject of e-mail or document with track changes in

relevant feed-back folder]or [meeting/workshop]

Feed-back

from:

filled in by

author(s)

Subject for QA QA-

person

[initials

or

name]

or [n/a]

Comments

Short comments should be

written in this form; longer

comments in e-mails saved

in the relevant QA-folder –

indicate in this column by

“e-mail” and subject +

date)

Document with

track changes

saved in the

relevant QA-

folder [no / yes +

version no.]

Date

of

QA

Changes

incorporated

[initials +

date +

document

version no.]

Technical QA

Quality of Information

(p. 80-82)

Validity and reliability

(p. 83-85)


Version Final

134

Subject for QA QA-

person

[initials

or

name]

or [n/a]

Comments







date)

Document with

track changes

saved in the

relevant QA-

folder [no / yes +

version no.]

Date

of

QA

Changes

incorporated

[initials +

date +

document

version no.]

Independence (p. 86-87)

Interdisciplinary

perspective (p. 88-90)

Evaluation ethics (p.91-93)

Is comparative advantage

of ITC covered?

Is performance of ITC

intervention covered?

Is capacity of ITC and

donor funding modalities

covered?

Others

Structure, language, etc.

Structure of report (p. 98-

99)

Editing (p. 100-102)

Layout


Version Final

135

Subject for QA QA-

person

[initials

or

name]

or [n/a]

Comments







date)

Document with

track changes

saved in the

relevant QA-

folder [no / yes +

version no.]

Date

of

QA

Changes

incorporated

[initials +

date +

document

version no.]

Others

Page numbers refer to Danida’s “Evaluation guidelines”

Paper versions of this document will be updated in one copy only.

Elaborated by: Controlled by: Approved by: Version: Date of approval:

AMS 1


Version Final

136

6.4.6 QA Annex 3 Template for Reports

List of contents

Error! No table of contents entries found.

1. First Title


Version Final

137

6.4.7 QA Annex 4 File Structure

Structure of the Filing System

Administration

o Budget

o Contracts

o Travel

o Minutes of meetings

o Activity Reports

o Expenses

Country Studies

o QA

o Draft

o Final

o Notes

o Documents

• Paper based documents

• Electronic documents

Desk Study Report

o QA

o Draft

o Final


Version Final

138

o Notes

o Documents



Evaluation Methodology

Evaluation Tools

Inception report

o QA

o Draft

o Final

o Notes

o Documents



Information on ITC

Templates

Terms of Reference


Version Final

139

1. Synthesis Phase

o QA

o Draft

o Final

o Notes

o Documents



2. Final Report

o QA

o Draft

o Final

o Notes

o Documents




Version Final

140

6.4.8 Structure of the WEB Server

(matches the filing system)

1. Administration

o Budget

o Contracts

o Travel

o Minutes of meetings

o Activity Reports

o Expenses

o Emails

2. Inception report

o QA

o Draft

o Final

o Notes

o Documents

3. Desk Study Report

o QA

o Draft

o Final

o Notes


Version Final

141

o Documents

4. Field Studies

o QA

o Draft

o Final

o Notes

o Documents

o Reports


Version Final

142

5. Synthesis Phase

o QA

o Draft

o Final

o Notes

o Documents

6. Final Report

o QA

o Draft

o Final

o Notes

o Documents

Joint Evaluation of the International Trade Centre documents/Different... · 2017-05-09 · Joint...

Documents

Transcript of Joint Evaluation of the International Trade Centre documents/Different... · 2017-05-09 · Joint...