Implementation Completion and Results Report (ICR) Document · 2018. 6. 5. · 07 04-Jan-2016...

Document of

The World Bank FOR OFFICIAL USE ONLY

Report No: ICR00004363

IMPLEMENTATION COMPLETION AND RESULTS REPORT

IBRD-81630-CN

ON A

LOAN

IN THE AMOUNT OF US$50 MILLION

TO THE

People's Republic of China

FOR THE

YUNNAN TECHNICAL AND VOCATIONAL EDUCATION AND TRAINING PROJECT ( P122008 )

May 24, 2018

Education Global Practice

East Asia And Pacific Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

Exchange Rate Effective April 11, 2018

Currency Unit = Chinese Yuan (CNY)

US$1 = CNY 6.27

FISCAL YEAR

July 1 - June 30

Regional Vice President: Victoria Kwakwa

Country Director: Bert Hofman

Senior Global Practice Director: Jaime Saavedra Chanduvi

Practice Manager: Harry Anthony Patrinos

Task Team Leader(s): Liping Xiao

ICR Main Contributor: Suzana Nagele de Campos Abbott

ABBREVIATIONS AND ACRONYMS

CBT Competency-Based Training

CPS Country Partnership Strategy

DVMS Dali Vocational Middle School

ECP Environmental Codes of Practice

FM Financial Management

GDP Gross Domestic Product

ICR Implementation Completion and Results Report

IRR Internal Rate of Return

JSS/SSS Junior Secondary School and Senior Secondary School

M&E Monitoring and Evaluation

MHRSS Ministry of Human Resources and Social Security

MOE Ministry of Education

MTR Midterm Review

NDRC National Development and Reform Commission

PAD Project Appraisal Document

PDO Project Development Objective

PIU Project Implementation Unit

PPMO Project Provincial Management Office

PV Present Value

QAS Qujing Agricultural School

TM Task Manager

TVET Technical and Vocational Education and Training

TVMS 1st Tengchong Vocational Middle School

YFTC Yunnan Forestry Technical College

YLRVC Yunnan Land and Resources Vocational College

YPFB Yunnan Provincial Finance Bureau

YTC Yunnan Technician’s College

YTS Yunnan Tourism School

YTTC Yunnan Transportation Technician College

YTVC Yunnan Transportation Vocational College

TABLE OF CONTENTS

DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED.

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5

A. CONTEXT AT APPRAISAL .........................................................................................................5

B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) ..................................... 10

II. OUTCOME .................................................................................................................... 11

A. RELEVANCE OF PDOs ............................................................................................................ 11

B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 11

C. EFFICIENCY ........................................................................................................................... 13

D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 15

E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 16

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 17

A. KEY FACTORS DURING PREPARATION ................................................................................ 17

B. KEY FACTORS DURING IMPLEMENTATION .......................................................................... 20

IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 23

A. QUALITY OF MONITORING AND EVALUATION (M&E) ......................................................... 23

B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE.................................................. 25

C. BANK PERFORMANCE ........................................................................................................ 25

D. RISK TO DEVELOPMENT OUTCOME .................................................................................... 27

V. LESSONS AND RECOMMENDATIONS ............................................................................. 27

ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 30

ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 39

ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 41

ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 42

ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 51

ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 52

ANNEX 7. SUCCESSFUL REFORM INITIATIVE HIGHLIGHTS IN YUNNAN TVET PROJECT ............ 53

ANNEX 8. CASE STUDIES ...................................................................................................... 54

The World Bank Yunnan Technical and Vocational Education and Training Project ( P122008 )

Page 1 of 55

DATA SHEET

BASIC INFORMATION

Product Information

Project ID Project Name

P122008 Yunnan Technical and Vocational Education and Training

Project

Country Financing Instrument

China Investment Project Financing

Original EA Category Revised EA Category

Not Required (C)

Organizations

Borrower Implementing Agency

People's Republic of China Yunnan Provincial Department of Education

Project Development Objective (PDO) Original PDO

To improve the quality and relevance of technical and vocational education and training to produce skills that respond to the labormarket demand, contributing to Yunnan’s economic development during China’s 12th Five-Year Plan period.


Page 2 of 55

FINANCING

Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$)

World Bank Financing IBRD-81630

50,000,000 50,000,000 50,000,000

Total 50,000,000 50,000,000 50,000,000

Non-World Bank Financing

Borrower 40,970,000 35,740,000 0

Total 40,970,000 35,740,000 0

Total Project Cost 90,970,000 85,740,000 50,000,000

KEY DATES

Approval Effectiveness MTR Review Original Closing Actual Closing

15-May-2012 02-Nov-2012 18-May-2015 31-Dec-2017 31-Dec-2017

RESTRUCTURING AND/OR ADDITIONAL FINANCING

Date(s) Amount Disbursed (US$M) Key Revisions

KEY RATINGS

Outcome Bank Performance M&E Quality

Highly Satisfactory Satisfactory Substantial

RATINGS OF PROJECT PERFORMANCE IN ISRs

No. Date ISR Archived DO Rating IP Rating Actual

Disbursements (US$M)

01 30-Sep-2012 Satisfactory Satisfactory .13

02 17-May-2013 Satisfactory Satisfactory .13

03 23-Dec-2013 Satisfactory Satisfactory 5.13


Page 3 of 55

04 23-Jun-2014 Satisfactory Satisfactory 10.62


06 26-Jun-2015 Satisfactory Moderately Satisfactory 18.12

07 04-Jan-2016 Satisfactory Satisfactory 24.37




11 01-Jan-2018 Satisfactory Satisfactory 47.82

SECTORS AND THEMES

Sectors

Major Sector/Sector (%)

Education 100

Public Administration - Education 1

Workforce Development and Vocational Education 99

Themes

Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 100

Jobs 100


Page 4 of 55

Human Development and Gender 102

Education 52

Access to Education 13

Science and Technology 13

Teachers 13

Standards, Curriculum and Textbooks 13

Labor Market Policy and Programs 50

Labor Market Institutions 25

Active Labor Market Programs 25

ADM STAFF

Role At Approval At ICR

Regional Vice President: Pamela Cox Victoria Kwakwa

Country Director: Klaus Rohland Bert Hofman

Senior Global Practice Director: Xiaoqing Yu Jaime Saavedra Chanduvi

Practice Manager: Luis Benveniste Harry Anthony Patrinos

Task Team Leader(s): Xiaoyan Liang Liping Xiao

ICR Contributing Author: Suzana Nagele de Campos Abbott


Page 5 of 55

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES

A. CONTEXT AT APPRAISAL

Context

1. At the time of appraisal of the Yunnan Technical and Vocational Education and Training Project (the project), China was one of the world’s fastest-growing economies with an average annual growth rate of 10 percent in the previous three decades. Average gross domestic product (GDP) per capita reached more than US$6,000 in 2012. Under the then recently approved National Economic and Social Development 12th Five-Year Plan (2011–2015), the Government of China (the Government) targeted annual GDP growth of 7.5 percent. The Government was keenly aware that to sustain that economic growth together with social development, a successful shifting of the economic development pattern from low to higher value-added industries, from export to domestic consumption, within a framework of inclusive growth and environmental conservation would be critical. Improving the quality and productivity of China’s labor was explicitly put forth as an enabling factor to the success of its new Five-Year Plan. In fact, the World Bank’s Investment Climate Assessment Survey identified worker skills and education as a constraint to China’s continued growth.

2. Faced with unprecedented growth, the Government was increasingly conscious of the value of highly skilled human capital, as reflected in its Medium- and Long-Term Talent Development Plan for 2010–2020, that pledged to increase the labor force skills and attract foreign talents, targeting for 28 percent of the labor force to be highly skilled by 2020. At the same time, the Education Development Plan (2010–2020) set goals and reforms to better align the education system with the demands of the economic and social development by expanding technical and vocational education at the secondary and tertiary levels.

3. Yunnan is a poor province located at the southwestern border of China. It is an important border province with abundant natural resources and a high population of ethnic minorities. Yunnan is a medium-size Chinese province, with a population of 45 million, 30 million of which reside in rural areas and 15 million in urban areas, and 84 percent of Yunnan’s land mass is mountainous. Yunnan further distinguishes itself from other provinces by its high level of ethnic diversity: only 66 percent of the population is ethnic Han, and the remaining one-third, or 15 million, is of the other 26 ethnic groups. Yunnan’s total GDP grew at an annual rate of more than 10 percent between 1980 and 2010, but compared with the rest of China and in particular to coastal provinces, Yunnan’s economy was less developed. Its GDP per capita in 2010 was only slightly more than half the national average, and the per capita net income of rural households in 2009 was only the equivalent of about US$530. Yunnan was the third poorest province in China—more than 10 percent (about 5.4 million) of its population was poor.

4. The Government had prioritized Yunnan’s development in its 12th Five-Year Plan and had formulated a strategy to boost its economic development by at least doubling the value of each of its pillar sectors’ (including agriculture, tobacco, tourism, mining, and energy) contribution to GDP. However, the structure of Yunnan’s economy was undergoing a fundamental transformation, as urbanization continued and the economic development pattern shifted toward higher value-added sectors. Demand was shifting toward a labor force in the secondary and tertiary industry and pillar sectors and unmet demand for these skills was increasing; bottlenecks were expected to become more evident, constraining growth. National data confirmed that the growth of GDP was creating proportionately fewer jobs, and those


Page 6 of 55

that were being created required more skills. Rates of return to education and income inequality had risen, signaling the demand for skills, as earnings of those without skills in demand were falling behind those with needed skills.

Sectoral Context

5. Technical and vocational education and training (TVET) in China is available primarily after nine years of basic education and tends to be formal with provision dominated by the public sector.1 Most students complete the nine-year basic education and then choose an academic, vocational, or technical secondary school, each lasting three years. Formal TVET in Yunnan is offered at four different levels: (a) junior secondary (comprising 14 junior vocational schools enrolling 6,900 students in 2010); (b) senior secondary (comprising 91 specialized secondary, 36 skilled worker, and 184 vocational high schools, (enrolling a total of 663,400 students in 2010); (c) tertiary (comprising 35 tertiary specialized colleges and 26 tertiary technical and vocational colleges, enrolling 183,300 students in 2010), and (d) adult technical and vocational education and training (enrolling 7,800 students in 2010). Only about 10–12 percent of the total TVET enrollment in Yunnan was in private institutions.

6. Although Yunnan had expanded access to the nine-year basic education to nearly all of its population and was rapidly expanding access to post-secondary, in 2010 the gross enrollment ratio for senior secondary (including TVET) was about 60 percent and for tertiary education it was about 18 percent; a significant portion of students did not have access to post-basic education. The National Medium- and Long-Term Education and Development Plan had set targets for Yunnan to reach a 90 percent gross enrollment ratio for secondary and 40 percent for tertiary education, half of which was expected to be on the technical and vocational track.

7. Apart from issues of enrollment capacity, technical and vocational education needed to be strengthened and transformed toward an efficient, equitable, and demand-driven system that could respond to the evolving needs of the economy and society. As it was, the system faced several challenges at the system and institutional level. First, the TVET sector was fragmented, with TVET institutions officially managed by the Ministry of Education (MOE), the Ministry of Human Resources and Social Security (MHRSS), and their local departments (provincial, city, and county levels), but with no single central agency responsible for TVET. Second, the levels of financing for education and training were low. Vocational and technical education is the responsibility of local governments that cover more than 80 percent of total public spending on education at all levels. Central government funds for TVET were extremely limited, with the result that households and businesses covered up to 40 percent of the total cost of education and training—twice the average share of private spending in other countries. Third, average enrollment in TVET institutions had increased rapidly, but the increase was not matched with improvements in education hardware and software, resulting in the lack of qualified teachers and overcrowding. The low level of expenditure per student in TVET schools seriously eroded schools’ capacity to provide quality instruction, up-to-date curricula, modern teaching, and learning facilities for quality TVET education. Fourth, fragmentation of the TVET sector further reduced the effectiveness and equity of public financing, due to varying funding mechanisms and standards. Schools managed by provincial-level departments were funded on per student allocation and tended to have a higher level of per student funding than those managed by the city- and county-level governments. Fifth, school-industry collaboration needed to be strengthened in a systematic manner, as despite experimental efforts to achieve this, schools’ capacity for planning and management to strengthen the link of schools and industry was weak, and there was a lack of incentive, culture, and legal framework for industries to provide systematic and sustained inputs to the management of TVET schools. Sixth, across all types of TVET programs, there was a lack of competency-based standards as the basis for curriculum development, and the standards developed by both

1 In China’s formal education system, the technical and vocational track becomes available at the junior secondary level after six years of primary education and continues to be offered at the senior secondary and tertiary levels. However, it is increasingly rare for students to proceed to junior vocational schools after primary school.


Page 7 of 55

the MHRSS and MOE (each issued separate standards) were developed with little inputs from the industries. Not only did the competency-based standards require updating, but the majority of TVET schools needed to revise their curriculum based on updated standards from industries. Finally, the quality of instruction and instructors needed to be upgraded. Teaching and learning observed in schools emphasized the instruction as a lecturer providing knowledge of theory but little hands-on experience with actual equipment and the production problems in a modern industry setting. Instructors needed to upgrade their theory and practice skills as few had both the pedagogical and technical skills needed to help students learn the use of new technologies.

Theory of Change (Results Chain)

8. The theory of change was that investments made under Component 1 would improve the quality and relevance of technical and vocational education to produce skills that respond to labor market demand in nine project schools. To raise awareness of the value of links with industry and the process of competency-based curriculum development and student-centered pedagogy, investments under Subcomponent 1.3 would provide training to school managers and teachers, to develop buy-in and ensure that they had the tools to implement project investments at the school level. Investments under Subcomponent 1.1 would establish institutionalized links between schools and industries and using these links to develop competency-based standards and to revise training programs responding to industry and market needs assessments. Utilizing the outputs from these, investments under Subcomponent 1.2 would develop and publish school-based curricula and related teaching learning materials. Teaching and student assessment systems, developed under Subcomponent 1.4, would ensure that teachers were implementing the new curricula appropriately, through student-centered pedagogy in which they were trained (under Subcomponent 1.3), and that students’ training responded to labor market demand. Investments under Subcomponent 1.5 would provide the necessary facilities and equipment to implement the new competency-based curricula utilizing student-centered pedagogy and to provide training to teachers. The project’s impact would be enhanced by allowing non-project schools to participate in training activities and have access to curricula and training materials developed by the nine project schools. Investments under Component 2 would support the carrying out of investments under Component 1 in an effective manner by providing the necessary coordination mechanisms and access to guidance, technical assistance, and feedback. For example, the technical assistance provided to schools under Subcomponent 2.2 was indispensable in supporting schools’ development of competency-based standards, curricula, and teaching and learning materials. Investments under Subcomponent 2.3 would support the documenting of lessons and cross-fertilization of project experiences to ensure a sustained impact on the nine project schools and provincial authorities engaged in TVET. By documenting TVET lessons and experience under policy briefs, allowing non-project schools to participate in training and workshops supported by investments under Subcomponent 1.6, the project would be expected to impact TVET at the broader provincial level, way beyond the nine project schools. A figure presenting the Project’s theory of change is presented in Annex 1.

Project Development Objectives (PDOs)

9. The PDO was to improve the quality and relevance of technical and vocational education and training to produce skills that respond to the labor market demand, contributing to Yunnan’s economic development during China’s 12th Five-Year Plan period.


Page 8 of 55

Key Expected Outcomes and Outcome Indicators

10. The project’s key expected outcomes were to be measured at two levels. First, the project would measure the improved capacity of the TVET institutions in delivering quality TVET, as reflected in new and improved standards and curriculum modules, more and updated training facilities, better trained teachers and administrators, and improved mechanisms for school-enterprise collaboration. The second-level PDO results would be measured by higher graduates’ skill certification rates and improved labor market outcomes, including through the following PDO indicators: (a) percentage of graduates finding initial employment in specialties in which they were trained; (b) starting salaries of graduates (average); (c) employer satisfaction with graduates; and (d) percentage of graduates obtaining both skills certificate and diploma.

Components

11. The project had two main components: (a) Improving the Quality and Relevance of TVET Programs in project schools, and (b) Strengthening Provincial Capacity in Coordination, Policy Development, and Monitoring and Evaluation.

12. Component 1: Improving the Quality and Relevance of TVET Programs in Project Schools (Total US$81.56 million, IBRD US$49.38 million). This component had six subcomponents, as described in the following paragraphs.

13. Subcomponent 1.1: Strengthen School-Industry Collaboration and Develop Competency-Based Training (CBT) Standards. This subcomponent was designed to address the lack of systematic and substantive school-industry collaboration and to address the lack of relevance in training programs. It aimed to strengthen and institutionalize the link between schools and industry by: (a) establishing formal School-Industry Collaboration Committees at the school level and sector-specific Industrial Advisory Committees at the departmental level2, and (b) developing competency-based standards to revise the selected training programs based on inputs from the School-Industry Collaboration Committees and the sector-specific Industrial Advisory Committees and following the scientific methodology of industrial and market needs assessments. The competency-based standards would be incorporated into the project’s revised training programs.

14. Subcomponent 1.2: Develop School-Based CBT Curriculum and Teaching Learning Materials. This subcomponent was designed to address the challenges related to the overly theoretical and outdated curriculum that was not sufficiently reflecting industries’ needs and a teaching methodology that was predominantly teacher centered. Based on the CBT standards and revised training programs developed under Subcomponent 1.1, this subcomponent would: (a) develop core curriculum standards; (b) develop modular curricula and other teaching learning materials through a cycle of piloting, evaluating, and revising; (c) publish school-based new curriculum materials, and (d) establish a school-level teaching and learning resource bank.

15. Subcomponent 1.3: Train School Managers and Teachers. This subcomponent was designed to address the challenges related to improving management and instructional capacity. For school managers, the project would support domestic and overseas training of top- and middle-level managers related to: (a) raising awareness of the importance and specific instruments to strengthen links with industry; (b) improving technical understanding of the process of developing CBT curriculum, and (c) evidence-based policy making and overall management capacity. For teachers, the project would support domestic and overseas training related to: (a) the development and use of competency-based

2 The committees would have no less than half representation from the enterprises, industrial associations, or other civil society entities.


Page 9 of 55

curriculum; (b) new student-centered pedagogy, and (c) skills training in relevant industries toward achieving high levels of both skills and academic qualification.

16. Subcomponent 1.4: Develop Student and Teaching Assessment Systems that reflect the new CBT curriculum and results-oriented approach to TVET. This subcomponent was designed to address the challenges related to student assessment being overly theoretical, not reflecting competencies required by the industry, and the overall lack of an effective teaching assessment system. Specifically, the project would support: (a) the development and implementation of new student assessment instruments, including a graduate tracer study and employer satisfaction survey, and (b) the development and implementation of new teacher and teaching assessment instruments.

17. Subcomponent 1.5: Upgrade Key Instructional Facilities and Equipment. This subcomponent was designed to mitigate the shortage of training centers and key training equipment in the project schools. The project would support activities to: (a) expand and upgrade facilities, including construction of training centers within existing school campuses of the selected schools; (b) provide key training equipment needed for the revised training programs and consultancies related to the design and supervision of works and goods contracts, and (c) develop school guidelines for the management of all training facilities and equipment.

18. Subcomponent 1.6: Support Disadvantaged Non-Project TVET Schools. This subcomponent was designed to enhance the project’s impact by allowing interested nonparticipating TVET institutions to participate at their own expense in some of the Project’s training activities and by providing them with access to the CBT curriculum materials developed by the project. Only two of the project’s nine schools included this subcomponent in the project design.

19. Component 2: Strengthening Provincial Capacity in Coordination, Policy Development, and Monitoring and Evaluation (Total US$1.0 million, IBRD financing US$0.5 million). This component had four subcomponents, as described in the following paragraphs.

20. Subcomponent 2.1: Strengthen the Provincial Coordination of TVET Affairs. This subcomponent was designed to strengthen overall coordination of TVET in Yunnan Province. The province had created an interdepartmental TVET Coordination Committee led by the Department of Education and with representatives from the Department of Human Resources and Social Security and other departments involved in the provision and management of TVET. However, the committee, whose purpose was to share information and minimize redundancies in spending and ensure TVET coverage, was not functioning regularly. The project would support the committee to: (a) organize study visits to provinces with similar coordination mechanisms; (b) conduct regular meetings to discuss TVET affairs, and (c) visit project schools.

21. Subcomponent 2.2: Support the School-Level Development of CBT Programs. This subcomponent was designed to provide guidance and technical assistance to project schools in developing and implementing CBT standards and curriculum. The project schools were under different departmental management, ranging from secondary to tertiary institutions, and had training programs with different focus. It was important for the province to provide guidance and technical assistance and training to ensure that schools follow a standard scientific methodology in the development of CBT standards, curriculum, and other teaching learning materials. In particular, this subcomponent would support: (a) the establishment and operation of a Provincial TVET Expert Team to guide the curriculum development in school; (b) teacher training in student-centered pedagogy; (c) teacher training in the development of CBT curriculum, and (d) teacher training related to development of curriculum cluster.

22. Subcomponent 2.3: Policy Development and Sharing of Knowledge and Experience. This subcomponent was designed for the project to have an expanded and sustained impact on TVET schools and policy development in TVET. The subcomponent would: (a) carry out policy research and produce policy briefs related to various TVET topics, and (b)


Page 10 of 55

organize provincial TVET workshops to exchange school-industry collaboration and curriculum development case studies and lessons learned.

23. Subcomponent 2.4: Project Management, Monitoring and Evaluation. This subcomponent was designed to support project management, including monitoring and evaluation (M&E) of its activities. Specifically, it would: (a) finance training of provincial- and school-level project staff in project management, financial management (FM), procurement, and M&E; (b) support provincial- and school-level procurement and FM activities, and (c) carry out project M&E.

B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE)

Revised PDOs and Outcome Targets

24. The PDO was not revised, nor were the outcome targets.

Revised PDO Indicators

25. The PDO indicators were not revised.

Revised Components

26. The Project’s components were not revised.

Other Changes

27. There was a minor reallocation of loan funds (less than 4 percent) among categories of disbursement in July 2015 to reflect more demand on teacher training and technical support for school reforms and less demand on civil works during the implementation period.

Rationale for Changes and Their Implication on the Original Theory of Change

28. The minor reallocation had no impact on original theory of change.


Page 11 of 55

II. OUTCOME

A. RELEVANCE OF PDOs

Assessment of Relevance of PDOs and Rating

29. The project is fully aligned with the current World Bank Country Partnership Strategy (CPS) FY13–FY16.3 It responded directly to the CPS’ Strategic Theme 2 (Promoting More Inclusive Development), under Objective 2.2 (Strengthening Skills Development Programs—Including for Migrants), to help China develop a workforce with the competencies needed for a more complex and dynamic economy. The CPS specifically highlights that the World Bank could provide assistance by “piloting reforms and investments that improve the quality and relevance of technical and vocational education (including modular, competency-based curricula, deepening ties with industry, and promoting life-long learning) and supporting systems to evaluate quality and guide future reforms,” basically restating not only the project’s objectives but also the very reforms it supported. Therefore, the relevance of the PDO is rated High.

B. ACHIEVEMENT OF PDOs (EFFICACY)

Assessment of Achievement of Each Objective/Outcome

30. The original PDO was to improve the quality and relevance of technical and vocational education and training to produce skills that respond to the labor market demand, contributing to Yunnan’s economic development during China’s 12th Five-Year Plan period. This section evaluates the outcomes against the project’s Results Framework. Details on project outputs are presented in Annex 1. The project was extremely successful and exceeded end-of-project targets for all outcome and intermediate outcome indicators ahead of schedule. It has succeeded in not only improving quality and relevance of TVET in the nine project schools, but it has put in place a framework for TVET in Yunnan Province, with schools enabled to explore and innovate different methods of school industry collaboration and mandated to disseminate and cross-fertilize different experiences and lessons among schools in the province, in the country and abroad.

31. Quality of technical and vocational education and training. This was to be measured by the percentage of graduates obtaining both skill certificates and diplomas. There was notable achievement, exceeding expectations, in improving the quality of TVET programs, as measured by the key indicator selected to measure progress. The overall percentage of students who obtained both skill certificates and diplomas in the nine project schools increased from 84 percent at baseline in 2011 to 96.5 percent in November 2017 (exceeding the target of 94 percent).

32. The school reform consultant carried out a study report to assess the impact of its assistance, especially in the quality aspects of the PDO. The main findings were that: (a) although leaders attending orientation training on current developments in TVET found it very informative, it was not until almost the end of the project, after they could see clearly its impact on teacher and student performance, that they acknowledged the importance and value of the training; (b) overall, on-the-job training activities covering modern TVET curricula construction methods, and the resulting curricula, are expected to have a long-lasting impact, and the new curricula documentation appears to represent a significant improvement in comparison with older texts; (c) foundation training, that is, training for curricula construction, for

3 World Bank, Country Partnership Strategy for the People’s Republic of China for the Period FY2013–FY2016, Report No. 67566-CN, dated October 11, 2012.


Page 12 of 55

student-centered, active pedagogy, improved the team’s competence and improved their methodology, should be included as part of curricula construction and not part of general training; (d) updated teaching and learning standards and related assessing and coaching of teachers are starting to make a positive impact on the quality of teaching and learning in at least eight of the nine project schools, and (e) in comparing the use of active pedagogy of a representative sample of project-trained teachers with a baseline group of untrained teachers, the trained teachers showed a much higher standard of performance, using significant appropriate teacher-student interaction with a mix of active methods covered by training. Eight groups of students (total 126 respondents) taught by project-trained teachers selected from classrooms observed for the study were presented with questionnaires regarding the quality of teaching. The main findings were: (a) 76 percent found that teachers clearly checked to see that a good classroom learning environment was established; (b) 42 percent found that teachers used different teaching and learning methods; (c) 86 percent responded that teachers knew their names; (d) only 2 percent reported that teachers used more than half the time lecturing from the textbook; (e) 84 percent found that teachers explained how they could best help themselves to learn and showed the useful ways of doing so, and (f) 92 percent responded that the methods used by project-trained teachers to help them learn were better than those used by most of their other teachers. The consultants’ findings conclude that their efforts have had a very positive impact on at least eight of the nine project institutions and have made a considerable contribution to quality. Further, they state that now, hundreds of staff can “carry out targeted labor market enquiries, manage job function analysis and develop occupational standards, construct effective TVET curricula, critically discuss aspects of learning psychology implement active pedagogy, support the development of competence, distinguish the various categories of assessment, carry out formative classroom observations or even write a structured lesson plan based around learning objectives.”

33. Relevance of technical and vocational education and training was to be measured by: (a) the percentage of graduates finding initial employment in specialties in which they were trained, and (b) employers’ satisfaction with graduates, as measured by satisfaction surveys carried out at baseline, midterm, and project completion. Although there was no rigorous controlled impact evaluation, there was also notable achievement, exceeding expectations, in improving the relevance of TVET programs, as measured by the key indicators selected to measure progress. The percentage of graduates from project schools finding initial employment within six months in the specialties in which they were trained increased from 58 percent in 2011 to 87.5 percent in November 2017 (exceeding the target of 77 percent). Employer satisfaction with graduates from project schools was measured by employer satisfaction surveys, monitored at the school level and consolidated at the provincial level. At baseline in 2011, 73 percent of employers were satisfied with TVET graduates from the nine project schools. By project completion (November 2017), employers’ satisfaction had increased to 92.7 percent (exceeding the target of 88 percent). The base salaries of graduates from project schools increased from CNY 884 at baseline to CNY 2,779 at completion, while the provincial average of graduates’ salaries ranged from CNY 1,778 and CNY 2,265 in 2017.

34. Strong, mutually beneficial connections were established between schools and industries. For the schools, the relevance of their training programs was improved to respond to industries’ needs. In return, for industries, the quality of TVET graduates was enhanced, which strengthened the pool of available talent by shortening their school-to-work transition period and building enterprise loyalty.

35. The school reform activities were directly responsible for improvements in the quality and relevance of TVET in all project schools. Specifically: (a) the new competency-based curricula were reviewed by experts at the school and provincial levels and are being used in the classrooms at the school level together with TVET teaching learning products that have been developed, including training plans, curriculum standards, textbooks, and learning manuals; (b) a student-centered teaching environment is evolving and taking root through project interventions, especially the high-quality training program delivered by the school reform consultant; (c) training facilities and equipment were updated to provide more and better practical training; (d) quality improvement approaches have been adopted, based on Teacher Standards


Page 13 of 55

and Evaluation Handbook and Student Assessment Handbook, and an objective-oriented comprehensive inspection system has been introduced; (e) the schools expanded reforms to other majors and some schools established mechanisms incorporating all good practices at the school level; (f) an extensive and innovative effort has been put in place and utilized to disseminate best practices and share experiences to project and non-project schools in Yunnan, in other provinces, and in other countries, and (g) an additional 121 non-project schools in Yunnan have engaged in school reform activities, along the lines supported by the project. A case study of the impact of the project’s achievements is provided in Annex 8.

36. The improvements in quality and relevance of TVET described above helped produce skills that respond to the labor market demand, contributing to Yunnan’s economic development during China’s 12th Five-Year Plan period. The project targeted TVET programs in key sectors specified in the National Economic and Social Development (NESD) 12th Five-Year Plan, that represent 39.4 percent of Yunnan’s economy, including Agriculture and Forestry, Mining, Manufacturing, and Tourism. Before the project, companies in Yunnan complained that difficulty in finding skilled workers was a constraint to business growth. In 2010-2011, only 35 to 45 percent of job openings were filled, and only 50 to 55 percent of job applicants were hired. The project increased enrollment capacity by 45 percent from 2010 to 2017. In 2013 an additional 3,767 students were admitted to project schools; by 2017 this figure had increased to 18,850. Enrollment capacity in project schools has increased by almost 30,000 students.

37. The project’s contribution to the economy in its early stages was through internships when students worked in partner companies that contributed to the economy. As implementation proceeded, project schools began increasingly graduating students benefiting from improvement to TVET quality and relevance, and the provincial economy benefited from the skilled training graduates by companies in key industries. More importantly, for recently graduating cohorts, 87.5 percent of graduates found employment in the specialties for which they were trained, and which the provincial economy needed, and 92.7 percent of employers were satisfied with the graduates of project schools that they hired. The project has already produced skills that respond to labor market demand, and also, given increased enrollment (which was not a specific project objective) will continue to contribute to the province’s future growth, especially since the key sectors the project has been supporting have been growing at an average rate of about 14 percent over the 2010 to 2015 period.

Justification of Overall Efficacy Rating

38. Overall efficacy is rated High based on the overachievement of the PDO, as measured by outcome indicators in its Results Framework and by its impact by engaging an ever-increasing number of non-project schools and feeding into policy at the provincial level through the extensive research and documentation that was prepared and conducted to cement its lessons and achievements.

C. EFFICIENCY

39. Both the project’s ex post economic analysis and its design and implementation experience imply that it was carried out with efficiency. An analysis of the project’s efficiency is summarized in the following paragraphs and described in detail in Annex 4.

40. An ex post cost-benefit analysis, prepared following the methodology followed at Appraisal, reveals that the investment in the Project yields a high, 10.52 percent return (against an estimated 7.89 percent at Appraisal). This result is a conservative estimate, due to unquantifiable benefits, conservative wage assumptions, and more importantly, to unattributed quality factors that affect the wage premium of the graduates. Further, a sensitivity analysis shows that this ex post result is relatively robust and holds under various alternative assumptions. While the project’s interventions did


Page 14 of 55

not specifically focus on access, the new lecture rooms and laboratory facilities have helped the schools address space constraints for training activities and the additional space allows the project schools to enroll more students. Almost all project schools saw a large increase in enrollment, which would not have been possible without the new facilities financed under the project. The increased training space also helped set the stage for the ensuing teaching and learning reforms.

41. The unit costs of project interventions were favorable, in comparison with provincial averages. Civil works account for the largest share of the overall project financing. The unit cost per square meter of teaching and learning space constructed for all nine schools under the project ranges between CNY 1,732.85 and CNY 2,666.71. This is below the provincial average CNY 3,300 (provided by provincial Development and Reform Commission) of similar school projects. The equipment procured was in line with the training disciplines offered by each project school. While most equipment procured was highly specialized and technical, a comparison of the unit cost of computers procured by the project schools, which is a relatively common purchase among the project schools and is more comparable, ranges from CNY 3,112 to CNY 7,000, which is in line with the price of computers procured in domestic projects. The computers procured by the Yunnan Forestry Technical College (YFTC) were slightly more expensive due to higher technical requirements for specialized software usage.

42. The project’s design and implementation contributed to its efficiency. An allocation of estimated and actual usage of project funding is provided in Table 1. Among the activities, civil works accounted for 60.6 percent of total project costs, equipment accounted for 26.3 percent of project costs, and curriculum reform took a share of 12.9 percent of total project costs. The funds were distributed to the nine project schools based on school size and demand. While the Project Provincial Management Office (PPMO) played a central role in coordinating activities by nine schools, the management cost accrued at the provincial level was only US$500,000 equivalent, accounting for only 1 percent of the total financing, maximizing the funds allocated to the beneficiary schools. Although actual counterpart funding fell short of amounts estimated at Appraisal, counterpart allocations for consultants’ services and training—expenditures that guided key school reform activities and helped promote overachievement of several targets—actually more than tripled. Project activities were implemented very efficiently and were completed six months ahead of the original closing date.


Page 15 of 55

Assessment of Efficiency and Rating

43. Efficiency is rated High based on its high estimated return, the reasonable unit costs of construction and other inputs, and the extremely high efficiency with which the project was implemented, with most activities completed well ahead of schedule.

D. JUSTIFICATION OF OVERALL OUTCOME RATING

44. The project’s overall outcome is rated Highly Satisfactory, based on its high relevance, its high efficacy, and its high efficiency and its impact that surpassed expectations by extending well beyond its target schools and fed into TVET policy at the provincial level, thereby cementing the model it introduced. Increased funding reallocated to teacher training and technical support on school reforms through consulting services together with decreased funding on civil works contributed greatly to exceeding end targets significantly for some intermediate outcome indicators. Although the project’s Implementation Status Reports (ISR) rated “Progress towards Development Objectives” as Satisfactory throughout its implementation, given information obtained since the final ISR, a rating of Highly Satisfactory is justified. This is because this ICR: (a) evaluates the project’s entire four and one-half year implementation, not just a specific six-month period of most ISRs; (b) evaluates overall outcome incorporating findings from an ex post economic analysis that confirms its high efficiency, and (c) benefits from new evidence on outcomes that has become available, and which it incorporates, namely the tracer and enterprise satisfaction surveys. Finally, the project represents a best practice example of the benefits of investing heavily and continuously in “soft-side” activities, namely school reforms and training, as opposed to civil works.


Page 16 of 55

E. OTHER OUTCOMES AND IMPACTS (IF ANY)

Gender

45. Although the PPMO did not compile disaggregated gender data, both female teachers and students benefited equally from the project activities. All project schools made great efforts, especially in providing more training opportunities to female teachers, recruiting more girls, and increasing their employment during project implementation.

Institutional Strengthening

46. The project had a strong impact on institutional development. Extensive capacity building for school administrators and teachers resulted in more double-diploma teachers becoming multitaskers who mastered the skills of CBT curriculum development and active teaching learning; in administrators being capable of carrying out evidence-based management and evaluation; and in schools, with teachers and administrators together, establishing a system of continuous updating for the curriculum syllabi and teaching learning materials to respond to the needs of industry. Based on policy research under the project, the PPMO prepared a policy note on Building a Modern TVET System in Yunnan and presented it to the Provincial Department of Education. This policy note fed into the province’s 13th Five-year Plan. Further, the project had a notable institutional impact through its emphasis on knowledge-sharing activities among the project schools and the non-project schools in the province. Also, quarterly briefing notes were circulated to the Yunnan Education Department, related Departments and all the project schools shared the project outcomes and lessons learned. Training under the project also strengthened project schools’ management abilities not only to carry out project activities but, more importantly, to institutionalize the curriculum development, delivery, assessment, and adjustment model that it supported.

Mobilizing Private Sector Financing

47. All project schools have established school-industry collaboration agreements. The model of school-industry collaboration established under the project led to several models of leveraging schools’ activities through the mobilization of private sector financing. Several project schools established company-specific training programs, whereby the companies invested in the schools by providing facilities, equipment, and training. In some cases, teachers were certified as enterprise training specialists. In other project schools, enterprises established facilities (hotels, jewelry workshops, and stores) that became training facilities for TVET students by providing flexible working and learning schemes with teachers and industry specialists working together. In all, about US$8.363 million was mobilized from private companies in support of the nine project schools in automotive services, tourism, and so on. A very successful example of school-industry cooperation that has mobilized significant amounts of private financing is presented in Annex 8.

Poverty Reduction and Shared Prosperity

48. The project had an important impact on poverty. More than 75 percent of the project school students that benefited directly from better vocational education and employment are from the rural poor. The average starting salaries of graduates of project schools more than doubled from CNY 884 at baseline to CNY 2,309 in 2017. Their earnings helped


Page 17 of 55

lift their families out of poverty. TVET schools in poorer areas were not able to participate in the project due to difficulties in providing counterpart funding. Therefore, the process of reaching out to partner schools through knowledge exchange and participation in training is likely to have had a poverty impact as well. Further, several project schools worked with disadvantaged local communities to provide training and assistance for their income-earning, community-based initiatives.

Other Unintended Outcomes and Impacts

49. The reforms in several project schools have been recognized through different accolades. Three out of nine project schools were upgraded from secondary to tertiary levels. One vocational college was selected by the Ministry of Education as one of the 100 piloting institutions for the national Modern Apprenticeship Program. Another project school was included in the list of the Top 50 Contributors in the Higher Vocational Education Quality Annual Report in China 2016 and two have been included in the following year’s report (2017) as among the top 50 schools with international influence. Several prizes were awarded to project schools in the provincial and national skills competition.

50. The Yunnan Department of Finance/Development and Reform Commission has appraised the project as the best World Bank-financed project in the province and will recommend it to the Ministry of Finance as a best practice.

51. With the increased capacity that resulted from infrastructure improvements and provision of experimental and practical equipment, student enrollment among project schools increased from 58,218 to 84,602 students during the project’s implementation period.

52. Some of the project schools have internationalized their academic achievements. In March 2018, delegates from four project schools participated in an international TVET conference in Dhaka, Bangladesh organized by the World Bank and signed 82 Memoranda of Understanding with schools in Bangladesh for collaborations in the areas of studying abroad, teacher exchanges, etc. Some project schools are also offering full scholarships for students from Bangladesh to study in China.

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME

A. KEY FACTORS DURING PREPARATION

53. The project was well prepared, with strong commitment from provincial authorities. Preparation began formally in March 2011 with an Identification Mission that ended with a four-day workshop with relevant school officials and teachers involved, to agree on the project concept, development objectives, main components and activities, M&E arrangements, and preparation schedule. At the time, the PPMO had been established under the Provincial Education Department and staffed with a Director, Deputy Director, and four other staff. The PPMO had selected the project schools based on predefined criteria,4 carried out rounds of training to help them prepare project proposals, organized a study visit to Singapore, and prepared at least two rounds of the project proposal. At the level of each of the nine project schools, a leading group led by the school principal had been formed and a working group of four to five staff assigned at the Project

4 The predefined criteria included links with the four priority sectors (bioindustry, automobiles, tourism, and geology and minerals), quality of the school proposal, and ability to repay the loan.


Page 18 of 55

Implementation Unit (PIU) to cover key functions in curriculum development, procurement, FM, and M&E. As a result, the quality of the province’s project proposal was very high despite its complexity.

54. Project preparation (and, later, implementation) benefited from the PPMO’s strong experience with World Bank-financed projects. The PPMO had been responsible for implementation of three earlier World Bank-financed projects,5 from which it had obtained rich experience with project preparation requirements, including project operations manuals, M&E frameworks and requirements, and World Bank FM and procurement guidelines and procedures and safeguard policies. As the project schools had limited, if any, experience with World Bank requirements and procedures, the PPMO’s experience was instrumental in supporting the nine schools in the preparation of their project proposals and consolidating them at the project level.

55. Following Pre-Appraisal in July 2011, the Project’s Appraisal Mission was launched with a workshop and was almost ready for implementation at the time of the workshop’s wrap-up meeting in September 2011. The workshop was led by the Provincial Deputy Director General of Education and included representative of the PPMO, the Provincial Department of Finance and Auditing, and key project staff from the nine project schools. The school-level project log frames, their results frameworks, procurement and training plans, the Project Administration Manual, and FM Manuals had been prepared, allowing for productive assessment, review, and suggestions by the World Bank’s appraisal team. Schools had submitted all the baseline data and established annual targets for each of the indicators in the results framework. The World Bank had conducted procurement training for staff of schools’ PIUs. The main changes during the Appraisal Mission related to the inclusion of a new Subcomponent 2.1 to help improve coordination of TVET in Yunnan. The Appraisal team also provided suggestions on the terms of reference for the Provincial Expert Team, with at least 50 percent membership from industry, that would substitute for what had before been dialogue activities with industry. Given the readiness for implementation, the Appraisal team recommended that up to 20 percent of the proposed loan financing be provided as retroactive financing for payments for eligible goods and services after October 2011.

56. The project’s objectives and design were fully aligned with the Government’s 12th Five-Year Plan (2011–2015), with its Medium- to Long-Term Education Development Plan (2010–2020), with its Medium- and Long-Term Talent Development Plan for 2010–2020 (Section I A), and with the World Bank’s CPS.6 Specifically, the project responded to the second objective (Building Capacity of the Poor to Participate in China’s Growing Economy) of the CPS’ Pillar 2 (Reducing Poverty, Inequality and Social Exclusion). The project focused on one of China’s poorest provinces. Further, it was estimated that more than 75 percent of students who would benefit from the Project would come from rural poor areas, and about 30 percent would belong to ethnic minority groups, thereby also aligning the project with the CPS’ Pillar 2 more broadly.

57. The project’s design benefited from lessons of experience in the design and implementation of two previous World Bank-financed TVET projects in China and of other Government-financed TVET initiatives. Specifically, these included improving TVET governance, building strong links between schools and industry, decentralizing school management, adopting modular, competency-based training, promoting life-long learning, building a learning culture, and offering pathways to advanced skills in tertiary education. All of these were addressed in the Project’s design, through the very activities it financed and its implementation modalities. It benefited from a background study carried out to scope the sector and issues in TVET in Yunnan. The project aimed to address explicitly its focus on improving TVET governance by supporting the established Provincial Inter-Departmental TVET Coordination Committee. Although the committee was not functioning regularly, the project would provide support to its operation and strengthening under Subcomponent 2.1 for it to address the fragmentation of TVET provision and improve coordination of TVET issues including policy, financing, and

5 The First, Fourth, and Fifth Basic Education Project. 6 The applicable World Bank CPS at the time was the Country Partnership Strategy for the People’s Republic of China for the Period FY2006–FY2012 and its related Performance and Learning Review.


Page 19 of 55

technical assistance. It also benefited from the analysis and policy recommendations of a World Bank analytical report that was under preparation at the time, “Developing Skills for Economic Transformation and Social Harmony—Yunnan Province.”7

58. The project’s design incorporated an effective mix of central coordination and standardization of implementation arrangements and procedures but with sufficient delegation of planning, implementation, and decision making to empower schools to adapt funding to their needs and to innovate. According to the project’s design, it would be implemented by the Yunnan Provincial Department of Education and the nine project schools, with the PPMO responsible for overall day-to-day responsibility for coordination implementation by the schools and executing provincial-level activities. A Provincial Expert Team would provide technical guidance to the PPMO and schools. Each of the schools would have a similarly structured PIU, staffed with technical experts and administrators responsible for the different aspects of school reform and project management. The project’s Implementation Manual, providing procedures for the project’s general administration, procurement, FM, environmental management, and M&E, would guide the project’s implementation. Further, the project, as designed, would be the first World Bank-financed TVET project in China to experiment with centralizing the development of competency-based standards. This was seen as having merits in terms of system impact and quality control but was subject to the risk of delays in curriculum reform due to delays in timely procurement of a qualified firm. On the flip side, the project provided project schools with the flexibility to define priorities, their own ‘projects’, innovate, and reach out to other non-project schools to disseminate experiences and expand the project’s impact.

59. The project’s design was limited to two components, one to be implemented by project schools and the other at the provincial level, and was appropriate to the achievement of its PDO and the province’s and schools’ capacity to implement while at the same time addressing concerns on school selection. The second component, although not explicitly captured by the PDO was nevertheless required to support its accomplishment. The project included activities to: (a) develop sustainable mechanisms for school-industry collaboration to constantly improve curriculum content and teaching methodology to reflect competency standard required by enterprise, and (b) cultivate results-oriented and evidence-based policy-making culture and behaviors among school leaders. Further, during Identification, the team had considered school selection in view of the national Government’s requirement that project schools be responsible for loan repayment. This requirement effectively resulted in those schools that most needed support due to poor conditions not being able to participate under the project. Therefore, the preparation team recommended that the PPMO and project schools incorporate activities that would support these non-project schools into their project proposals, thereby replicating the project’s impact.

60. The identification of risks, the mitigation measures put in place, and the overall risk rating were all appropriate. The project’s overall risk rating at Appraisal was Moderate, with the main risks related to project schools not having had prior experience with World Bank-financed projects. These risks were to be mitigated by carefully designed training in financial management, procurement and monitoring and evaluation at the launch workshop and continuous technical support from the World Bank team during implementation.

7 World Bank, Developing Skills for Economic Transformation and Social Harmony (P123895), Report No. ACS3321, dated May 30, 2013.


Page 20 of 55

B. KEY FACTORS DURING IMPLEMENTATION

61. The US$50 million World Bank loan for the project was approved by the World Bank’s Board of Directors on May 15, 2012, was signed on August 6, 2012, and became effective on November 2, 2012. The project was never at risk throughout implementation. By the time of the midterm review (MTR) from May 18 to 23, 2015, the project was well on target to meeting its PDO and was exceeding targets for the key results indicators. Feedback from stakeholders (managers, teachers, students, and industry representatives) showed their support to school-based reforms and innovations, which were substantiated by the classroom observations and M&E results. The Yunnan Department of Finance commended the project’s implementation as one of the best in Yunnan Province. Project activities, which were expanded to beyond those originally planned at Appraisal, were completed well ahead of the loan’s original closing date of December 31, 2017. Several factors during implementation affected the project’s stellar performance in general and contributed to the achievement of its PDO, which exceeded expectations.

62. The project was ready for, and had in effect begun, implementation before effectiveness of the World Bank’s loan. The first implementation support mission, in January 2013, reported that the Provincial-wide Expert Group (with the mandate of advising, M&E, and disseminating good practices during the school curriculum reform process) had been established, all nine schools had mobilized and established School-Industry Collaboration Committees and sector-specific Industrial Advisory Committees, a final Procurement Plan had been approved, schools had prepared their 2012 Progress Reports and 2013 Annual Plans, and school reforms were under way. Two contracts (impact evaluation and a works contract) had been signed, and the bidding process of the contract for the core of school reform—developing standards and curriculum for four key sectors in automobile, tourism, agriculture, and mining—was under way. The early signing of the school reform technical assistance contract was important for keeping the project’s implementation on schedule. Any delay in signing this contract would have implied delays in implementation in each of the schools’ reform plans given the province’s proposal to centralize the development of competency-based standards and curriculum. Nevertheless, four schools eventually decided not to rely upon the overall consulting package and instead carried out curriculum development on their own.

63. Project management was strong and effective, and the institutional arrangements for implementation were sound. The PPMO performed extremely well in coordinating project activities with project schools’ PIUs, their governing authorities, other relevant departments at different levels, consulting firms, and the Provincial Expert Team. The Project Management Guidelines and the Financial Management Guidelines on Curriculum Reform were produced and updated routinely. At schools, PIUs were leading project implementation, under the direction of the Vice Presidents in charge of Teaching Affairs. The PPMO, together with the PIUs, prepared and submitted annual progress reports and annual plans as required. The PPMO visited project schools frequently to identify and address any emerging problems. The PPMO mobilized resources to engage eight expert teams to provide technical assistance on curriculum and pedagogy reform, M&E, and dissemination. Engaging consulting firms in implementation was an innovation in project management, and it put higher pressure on the PPMO’s coordination and monitoring capacity; additional staff were brought in to assist the PPMO. A pool of experts was established to facilitate the school-level curriculum development, which comprised outside experts and two experts from each project school. They inspected the schools regularly and provided feedback and guidance based on the classroom observation, meetings, and interviews with the teachers. The PPMO coordinated well with the project schools on the development of CBT programs at the school level mainly through the consulting firms and the Provincial Expert Team. The Provincial Expert Team supported the PPMO and PIUs through semiannual surveys, quarterly briefing notes, and progress reports. They played an active role in guiding the project schools to summarize the successful experience and lessons learned with a detailed outcome that followed the project log frame. The team also participated in the trainings delivered by the consultants and collected feedback of the trainees and provided follow-up technical support to all project schools.


Page 21 of 55

64. School-Industry Collaboration Committees and sector-specific Industrial Advisory Committees were instrumental in building connections with industries and improving the relevance of TVET. As a result, deepened school-industry collaboration contributed to the improved quality of graduates and an increased demand from employers. The project schools continuously explored innovative and effective collaboration models with enterprises, such as internship programs with flexible working and learning alternation. Internships of varying durations, with students paid by enterprises, were provided to almost all the students. Different forms of collaboration were introduced, including developing concrete school-industry projects such as industry-ordered training, agreements to use industries as training bases for students and teachers, and bringing factories to establish branches in school campuses. Annex 8 presents three types of cooperation models that project schools adopted. With the operation of new buildings and equipment, more joint centers could be set up with selected enterprises on campuses. The School-Industry Collaboration Committees and Industrial Advisory Committees routinely engaged in the development, review, and revision of the new CBT standards, curriculum, and other materials, by reaching out to industries to conduct in-depth market surveys of skills, involving relevant industrial experts in developing outcome-based standards and curriculum, and actively engaging the review of newly developed curricula at the school level, thereby providing validation to the standards and curriculum. The formalization of school-industry relationships through both of these groups of committees was instrumental in improving the quality and relevance of TVET programs in project schools and involvement of the private sector.

65. The PPMO engaged many consulting firms providing technical assistance, and one firm focused on the key school reform activity of CBT and student learning. The firm provided assistance for: (a) general training covering all nine project schools, providing the foundation upon which other activities would be implemented; (b) support and capacity building of curriculum construction teams in five of the nine project schools; (c) support for the development of updated teaching and learning standards, and (d) competency-based teacher training covering active pedagogy for seven project schools (added to the contract at a later stage). General training included: (a) orientation training for institution leaders to engage them on the benefits of student-centered pedagogy; (b) foundation training to support curriculum construction, to provide skills to carry out job function analysis, occupational standards development, and cost-effective labor market enquiries and to work effectively with industry subject matter specialists; (c) foundation training for student-centered pedagogy; (d) initial development of the standards documentation, which led to the development of a Teacher Standards and Evaluation Handbook and a Student Assessment Handbook, and (e) inspector training, to establish effective in-house teacher inspection programs following the standards documentation. Curriculum construction involved working with curricula construction teams to create curricula standards, detailed content, textbooks, and support materials. Updated teaching and learning standards involved piloting the Teacher Standards and Evaluation Handbook and Student Assessment Handbook. Finally, competency-based teacher training was provided to about 630 teachers in all project schools through two modes of training. The first engaged up to 30 teachers for two weeks of training and two classroom observations. The other engaged up to 80 participants for only one week of training. These trainings identified and trained high-potential teachers to become co-facilitators. Teachers who had benefited directly from training by the consultant later became teacher trainers and, with an increased number of training facilities, delivered cascade training at the school level. Four project schools have instituted in-house cascade trainings in their schools, and the others have expressed their intention to do so.

66. The project aimed to involve the Yunnan Interdepartmental TVET Coordination Committee to improve the provincial coordination of TVET matters, but this proved difficult so the PPMO assumed the responsibility for coordination very effectively. The TVET Coordination Committee did not function as the busy schedule of the leaders made regular meetings difficult. Most of these committees in other provinces have been cancelled by the Government for the very same reason. Recognizing the importance of coordination, the PPMO made great efforts to propel the provincial coordination, policy development, and M&E. It did so very effectively. For example, when counterpart funding issues arose for some schools, the PPMO met with related provincial and city authorities to resolve the issue. The PPMO also assumed a central role in the dissemination of project outcomes at the provincial level. Effective coordination of TVET affairs at the


Page 22 of 55

provincial level by the PPMO served to propel project implementation, with routine briefing notes on project progress and achievements provided to the Yunnan TVET Coordination Committee. Under the PPMO’s coordination, over 100 research papers were prepared to cement knowledge, including a policy note on building a modern TVET system in Yunnan that was presented to the Provincial Department of Education and fed into the Province’s 13th Five-Year Plan’s TVET policy strategy.

67. Training under the Project was intensive and continuous and exceeded expectations given innovations in delivery methods. International and domestic training programs were designed and delivered to build the capacities of school managers and instructors on school-industry links, the development and implementation of CBT programs (including training on student-centered methodology and curriculum development), and evidence-based policy development and management. The PPMO worked hard to organize two overseas trainings for administrators. The PPMO also provided 268 trainings on project management to the project schools. A total of 16,477 participants (person times) engaged in various training programs by the school reform consultant. As training was an important aspect of curriculum reform, schools also provided various training programs including lectures, practice in enterprises, and study tours. Project schools developed medium-term school-based teacher training plans, developed different training programs for different beneficiary teachers based on their needs, emphasized industry-based training, and conducted participatory training such as practice teaching under the guidance of veteran teachers. Overall, the trainings for school managers and teachers exceeded targets; in total, 310 school managers and 421 teachers (person times) were trained. As many as 103 school-based training events were organized. The number of overseas training programs and study tours surpassed original targets, with 31 short-term teacher groups visiting 16 countries, and 17 administrative staff groups visiting 20 countries. The capacity of school managers and teachers has been improved. In fact, by midterm, the percentage of teachers with both the skill certificate and academic qualification reached 69 percent, which exceeded the end target of 65 percent. In addition, 785,401 participants (person times) benefitted from the short-term training programs offered by the project schools to support local communities.

68. Schools progressed at different paces with the competency-based curriculum reform. Five project schools opted to count upon the support of the school reform consultant in curriculum development and related training. The flow path or procedures for developing demand-driven curricula was established starting with: (a) labor market surveys for the pilot specialties carried out by teachers to assess labor market needs and the development of competence-based training standards based on the survey results; (b) development or update of textbooks and teaching materials, and (c) pilot of the new textbooks and teaching materials in the classroom. The Provincial Expert Panel reviewed all the newly developed curricula, textbooks, teaching, and learning materials produced in the nine schools. The four project schools that did not seek technical assistance from the school reform consultant encountered greater difficulties in curriculum development, although they were able to complete the process.

69. Support to disadvantaged non-project TVET schools exceeded expectations, providing a catalytic platform upon which the Project’s reforms and activities could be replicated. To address concerns that the disadvantaged schools would not be able to participate under the project due to funding concerns, project design had included provisions for two project schools to allow non-project schools to participate in their training activities and provide them with access to their CBT and curriculum materials. The two designated project schools, Dali Vocational Middle School and Yunnan Forestry Technical College, each engaged a total of 15 partner schools in their student-centered training. Following their examples, all project schools took initiatives to engage partner schools.

70. Knowledge sharing was continuous and helped effectively cross-fertilize project schools’ experiences not only among themselves but to non-project schools in Yunnan and in other provinces in China and to other countries as well. The PPMO organized several workshops with the participation of provincial authorities, all project schools, consultants, and related industries to distill and disseminate good practices and lessons in school reforms, including the new curricula, standards, textbooks, other learning materials, and research reports produced under the Project. Knowledge-sharing


Page 23 of 55

activities were organized among programs, schools, and partner schools through reports, briefing notes, workshops, domestic and international study tours, and so on. Experiences with the reforms under the project gradually took root in the nine project schools after three years of implementation, and each school formulated and summarized their own experiences and lessons learned that were worthy of dissemination. Several project schools carried out international cooperation projects, and one actually established an office of international cooperation. A formal cooperation agreement was put in place with Bangladesh. The PPMO also shared their lessons learned with the Project Management Offices of the Xinjiang and Gansu TVET projects. Upon completion, the PPMO organized a knowledge-sharing workshop with broad participation from: (a) project schools in Yunnan, Xinjiang, Gansu, and Guangdong TVET Projects; (b) representatives of the MOE; (c) related provincial authorities, and (d) a Bangladesh delegation composed of central authorities of vocational education. Some non-project schools in Yunnan and national and international TVET experts also participated. Based on participant feedback, this workshop was highly successful (4.7 on a scale of 1 to 5). Recently, the project schools signed agreements with 82 TVET institutes in Bangladesh during the International Skills Workshop in Dhaka on March 8–9, 2018; therefore, the best practices produced from the project will be further disseminated to TVET institutes in Bangladesh.

71. The project’s implementation arrangements with delegation of planning, implementation, and decision making to project schools were successful in incentivizing schools to innovate and find solutions that met their particular needs. Project schools developed innovative industry-school cooperation arrangements, advanced curriculum reform to other majors, expanded and adapted support to partner schools, modified capacity-building arrangements, sought international partners, and implemented various other innovative solutions that matched their abilities, markets, students, and partners. A partial list of successful reform initiatives by project schools is presented in Annex 7.

72. Progress with civil works was satisfactory, but two schools faced issues that delayed the start of construction. Among the eight project schools that planned civil works, three had new buildings in operation by October 2014, three had buildings under construction, and the remaining two had not started construction, due to issues with counterpart funding and issues with relocation and approval of land use rights. With the intervention of the PPMO, these issues were resolved satisfactorily in January and May 2016, respectively, and construction proceeded without further delays. All civil works was completed satisfactorily.

73. Teachers played a very central role in the project both as contributors on development of links with enterprises, standards, curriculum and training design and delivery, and as beneficiaries of training, but their efforts were extracurricular and were not compensated. Teachers who participated in curriculum design and training were asked to do so over and above their normal duties as teachers. Nevertheless, according to local legislation, they could not be compensated for this extra service monetarily. The World Bank suggested repeatedly that in view of this constraint, it would be important to explore possible nonmonetary compensation, or incentives, to reward them for their efforts. Undoubtedly, the capacity of those teachers who participated in reform activities was improved. In fact, some of the teachers who participated in reform activities were promoted to key teachers and won awards in the city-, provincial-, and national-level teaching competitions.

IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME

A. QUALITY OF MONITORING AND EVALUATION (M&E)


Page 24 of 55

M&E Design

74. The project’s M&E aimed to monitor the progress and achievement of the PDOs through: (a) regular/routine monitoring to track progress of project activities against plans; (b) a MTR to assess early results/effectiveness of project activities, and (c) an impact evaluation of project outcomes at project completion to assess the project’s effects and the extent to which its final development objectives have been obtained and identify lessons on interventions that either contributed to or hindered their achievement. M&E activities, including training for M&E, were part of the Project’s design under Subcomponent 2.4.

75. The project’s results framework was well designed and appropriate to measure progress toward its PDO. The intermediate outcome indicators were linked to each of the project’s components and subcomponents and were well defined, with baseline and targets provided. Most data measuring outputs and intermediate outcomes were to be collected through project schools’ administrative sources and aggregated at the project level by the PPMO for analysis and input to subsequent stages of activities. Early in implementation, the PPMO procured a consulting firm to conduct the project’s evaluation, including validating baseline and actual values of all PDO and intermediate indicators using survey methods. The consultant would collect data through surveys, including graduate tracer, enterprise satisfaction, and school internal impact surveys, and verify self-reported data from project schools for all output and outcome indicators.

M&E Implementation & Utilization

76. Project schools routinely updated progress toward project indicators, and these fed into routine follow-up by the PPMO to address issues that were affecting expected progress (that is, counterpart funding affecting construction) and into a results-driven MTR. The consultant carried out project M&E by collecting data through surveys. The graduate tracer study, the enterprise satisfaction survey, and the school internal impact survey were conducted in 2014. They also verified self-reported data from project schools for all output and outcome indicators. Although numerous data and evidences were collected, the quality of the services provided by the M&E consultant fell short of expectations. There were several issues with the data and analysis utilized by the consultant—the sample size was too small, sampling methods had issues, comparator schools were not appropriate in terms of size and environment, and indicators to measure enterprises’ satisfaction were not appropriate and did not follow the indicators in the project’s results framework. Both the PPMO and the World Bank’s implementation support team observed that the capacity of the M&E consultant was weak from the beginning and provided regular feedback to the consultant, but it was not possible to terminate the contract and, in the end, the quality of the impact assessment the consultant produced was belo

Implementation Completion and Results Report (ICR) Document · 2018. 6. 5. · 07 04-Jan-2016...

Documents

Transcript of Implementation Completion and Results Report (ICR) Document · 2018. 6. 5. · 07 04-Jan-2016...