Sustainable use of products & tools in NTB province Survey ...Sustainable use of products & tools in...

Sustainable use of products & tools in

NTB province Survey results 2010 – 2011

Consolidation Programme Health / Policy Analysis & Formulation in the Health Sector (PAF)

Dr. Lieve GoemanDr. Rahmi SofiariniAnwar FachryMaddi Djara

With contributions of Dr. Paul Rueckert, Dr. Harmein Harun and Karsten van der Oord

Sustainable use of products and tools in NTB province

Sustainable use of products and tools in NTB provinceSurvey results 2010 – 2011

Consolidation Programme Health / Policy Analysis & Formulation in the Health Sector (PAF)

Dr. Lieve GoemanDr. Rahmi SofiariniAnwar FachryMaddi Djara

With contributions of Dr. Paul Rueckert, Dr. Harmein Harun and Karsten van der Oord

Download at: www.ighealth.org

The Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH was formed on 1 January 2011. It brings together the longstanding expertise of the Deutscher Entwicklungsdienst (DED) gGmbH (German development service), the Deutsche Gesellschaft für Technische Zusammenarbeit (GTZ) GmbH (German technical cooperation) and InWEnt – Capacity Building International, Germany.

For further information, go to www.giz.de


Contents

1. EXECUTIVE SUMMARY 5

2. INTRODUCTION 8

3. METHODOLOGY 9

3.1. CONCEPTUAL FRAMEWORK 9

Products 9

Health Facilities 9

Use of Products 10

3.2. SAMPLE 11

3.3. SURVEY TEAM 14

3.4. DATA SOURCES AND DATA COLLECTION INSTRUMENTS 14

3.5. DATA ENTRY AND MANAGEMENT 15

3.6. DATA ANALYSIS 15

3.7. FOLLOW-UP SURVEY (FUS) 16

4. RESULTS 17

Extent and quality of the use of the products (scores) 17

4.1. BREAKDOWN PER PRODUCT AND LEVEL OF HEALTH FACILITY 18

The products in detail 22

WISN 23

HRIS 24

HMIS 24

Referral System 25

IHPB 26

Integrated Monev 26

P/DHA 27

Desa Siaga 28

QI-Action 28

1


4.2. BREAKDOWN PER DISTRICT & GEOGRAPHICAL DISTANCE 29

BETWEEN HEALTH CENTERS/VILLAGES & THE DHO

4.3. INTEGRATION OF THE PRODUCTS INTO PLANNING 32

AND BUDGET DOCUMENTS

4.4. INPUT FOR PLANNING, ACTION AND POLICY CHANGE 35

4.5. QUALITATIVE RESULTS 36

Success Factors 36

Failure factors 36

4.6. LIMITATIONS OF THE SURVEY 38

5. INTERPRETATION OF THE RESULTS 40

Too many Failure Factors 40

Fear for change 41

Complexity of sustainability 42

Looking for perfection 42

Minimal to no differences between BLS and FUS 43

6. LESSONS LEARNT 44

7. CONCLUSION 45

8. ABBREVIATIONS 46

2


3


4


1. Executive Summary

BackgroundDuring 2006-2009, SISKES (Strengthening of District Health Systems) & HRD (Human Resources

Development) Projects supported the province Nusa Tenggara Barat (NTB) in strengthening district

health systems through improvement of health system management, health services management,

clinical services quality, human resource development and community empowerment. More than 200

products, tools, guidelines, information systems, manuals and films were developed to support the

achievement of the projects objectives.

In 2010 both projects were merged into 1 Consolidation Program Health: Policy Analysis and Formulation

(PAF) to consolidate and further strengthen the implementation and sustainable use of these products.

The program focused on policy efforts to create legal frameworks and the necessary laws and regulations

to institutionalize and mainstream the products. One of the indicators for measurement of the overall

objective that “Health policy guidelines and implementation regulations are better geared towards the

requirements of a decentralized health care system” is that “A representative number of health facilities

(at least 80%) in all 10 districts in NTB confirm that they implement or use the guidelines supported by

the project”.

MethodologyA baseline Survey (BLS) was conducted in May 2010 and a Follow up Survey (FUS) in April 2011 in

collaboration with the provincial health office (PHO) and a local research center in NTB. The nine most

important products in terms of investments and contribution to achieve the Maternal and Neonatal Health

strategy (AKINO) of NTB have been selected. Their use was assessed in 459 health facilities through

a cluster sampling method resulting in a representative sample for the province and this through 776

interviews with key respondents familiar with the projects and products. Their perceptions and answers

(subjective values) were cross checked with available proof and evidence (objective values). The “Use

of the products” was assessed at four “levels of use”: the availability and existence of the product in

the health facility; the implementation of the product and existence of all required conditions enabling

correct use; the incorporation of the use of the product into planning and budget documents; and policy

change or action obtained by the use of the product. SPSS v15 was used for the quantitative analysis.

5


The qualitative analysis focused on factors that influenced the use of products, and on the changes and

actions that resulted from the use of the products.

ResultsBoth surveys were able to demonstrate the availability and presence of the nine tools in the health

facilities (BLS 52.9 % and FUS 55.6%). When the quality and the extent of use were brought into

account with a scoring system, the results dropped to 25.8 for BLS and 26.4 for FUS. No significant

differences were noticed between BLS and FUS. The products with the highest scores were Village Alert

system (Desa Siaga), Health Resource Information System (HRIS) and Integrated Health Budgeting

and Planning (IHPB). Use of the products was the highest at District Health Office (DHO) level. Central

Lombok and Sumbawa were the two districts with the highest scores.

The surveys identified multiple factors which influenced successful use. Even though the products

provided valuable data and information and staff perceived the benefits of using the products in their

daily work, the quality of using the products and the degree of incorporation into policy were limited. Main

obstacles, identified by key staff, were absence of legal frameworks to make the use of the products

obligatory, commitment of decision makers, frequent staff rotation without proper handing over and

insufficient budget allocation.

InterpretationSISKES and HRD projects allocated high amount of resources in terms of time, money, human resources

and expertise to the developments of products and created the perfect conditions for sustainable use of

the products. This resulted into very positive and impressive results of coverage and use at the end of

the projects. Despite all these efforts the target of the Consolidation Program PAF e.g. 80% of all health

facilities are using the products, has not been achieved. The presence of too many failure factors, fear

for change, the complexity of sustainability and the effect of development agencies with high amounts

of funding, staff and expertise during a short period of time were the underlying reasons to explain the

disappointing results of the surveys. The consolidation phase focused during the past 1.5 year on policy

advice to mainstream the products in the health system but it seemed too short to ensure impact on the

use of the products.

Lessons learnedAll influencing factors should be considered and addressed. All conditions and requirements for

successful use of products have to be fulfilled. Without perceived benefits a product will not be used.

A health system with weak institutional and management structures will not facilitate the full integration

6


and use of a new product and cannot ensure its sustainability. Commitment from high level decision

makers is needed for successful use of products. A period of less than two years is too short to create

legal structures for the institutionalization of products. This has to be built in from the start of product

development and design together with ownership. Products developed during projects have many

dangers: they are often agency (donor) driven in the need for quick, tangible and visible results, they

may create duplication and add to the administrative burden and workload of staff. Newly designed

products have to be compatible with existing formats and systems. At the same time they have to be

easily changeable and adaptable to new requirements of decision makers or they will be abandoned.

7


2. Introduction

During 2006-2009, SISKES & HRD Projects supported NTB province in strengthening district health

systems through improvement of health system management, health services management, clinical

services quality, human resource development and community empowerment. More than 200 products,

tools, guidelines, manuals and films were developed to support the activities, to bring in norms and

standards and to reach the objectives of these projects. In 2010 both projects were merged into 1

Consolidation Program Health: Policy Analysis and Formulation (PAF) to consolidate and further

strengthen the implementation and use of these products.

The consolidation phase had as objective that “Health policy guidelines and implementation regulations

are better geared towards the requirements of a decentralized health care system”. One of the indicators

for measurement of the progress and achievement of the objective was that “A representative number of

health facilities (at least 80%) in all 10 districts in NTB confirm that they implement or use the guidelines

supported by the project”.

Therefore a baseline Survey (BLS) was conducted in May 2010 and a Follow up Survey (FUS) in

April 2011. The nine most important products have been selected to investigate the sustainable

use of products after the end of the projects. The survey answered the following questions: Did the

consolidation phase contributed to sustainable use of the products by offering limited technical advice

to finalize the implementation and by focusing on policy advice to support institutionalization of these

products at provincial and national level (mainstreaming)? What were the factors that influenced the

use? Which of the selected products had the best results, which district and at which level of health

facility are products best used?

8


3. Methodology

3.1. Conceptual Framework

To be able to answer the indicator and research questions a conceptual framework has been developed

that functioned as a “theoretical map” to ensure a systematic and comprehensive way in obtaining

objective unbiased results. This is presented in Table 1 below.

Products The nine most important products have been selected. Each of them supported the Maternal, Neonatal

and Child Health (MNCH) strategy of the province, namely “AKINO” or “No maternal death in the village”

and contributed to the improvement of the management of and quality of services in the health system.

• WorkloadIndicatorsStaffingNeed(WISN): a tool to calculate staffing needs based on workload

indicators

• HumanResourceInformationSystem(HRIS): a database based on MS Access which enables

HR Management and Planning

• HealthManagement InformationSystem (HMIS): A “one gate data flow” health management

information system

• Referral System:a technical guideline to establish the referral system, from referring patients

using referral and counter referral letters, to referral of specimen and transfer of knowledge

• Integrated Health Planning and Budgeting (IHPB): a guideline to develop and conduct the

process of integrated health planning and budgeting using existing data from HMIS

• IntegratedMonev: a guideline to conduct joint monitoring at district and province level, to be used

as input for the next planning cycle

• District and Provincial Health Account (P/DHA): public health expenditures at district and

provincial levels

• VillageAlertSystem(DesaSiaga): a comprehensive toolkit for raising community empowerment

and participation in reducing maternal and infant death in the village

• QIAction: an effort to improve quality of health services by focusing on identified problems at a

specific unit in the hospital

Health FacilitiesFive levels of health facilities, institutions and offices were identified: The Provincial Health Office

9


(PHO), the provincial hospital, the District Health Office (DHO), the district hospital, the subdistrict

Health Centre (Puskesmas), the sub health Center (Pustu, Polindes) and the village. The application of

each of the nine products has been verified at the appropriate level. For example HRIS was only used

at hospital, PHO and DHO.

Use of ProductsThe “Use of the products” was composed out of four “levels of use” in a hierarchical way. For each

product, firstly the availability and existence of the product (level 1) in the health facility was verified

followed by the implementation of the product (level 2). Then the incorporation of the use of the product

into planning and budget documents (level 3) was checked and if the product had led to policy change

or program action (level 4). The degree or extent of achievement of a higher level of use could not

be higher than the previous level. The argument is as follows: A product could not have led to action

and change if it was not used completely and producing the necessary results required for change. A

product had to be present in the facility before it could be used.

Table 1. Overview Conceptual Framework

No Product User:level of health facility

Availability or pres-ence of product in the health facility (level 1)

Use and implementa-tion of the product(level 2)

Incorporation into Planning and budget documents(level 3)

Action or change induced by the product(level 4)

1 WISN PuskesmasHospital

Guideline and docu-ment with the WISN results are available.

Report and proposal on personnel issues eg. HR mobilization, is made, based on the WISN results

WISN is stated in the planning document of the health facility (RKA).

Documentation of change, decree, local laws.

2 HRIS Hospital DHOPHO

• HR Information is available.

• Hardware and soft-ware are present.

Databank is complete and functioning.Proposal on per-sonnel issues eg. Training and HR mobilization is made based on the data from HRIS. Virus scan and up-dates are done.

HRIS is stated in the planning doc of the health facility (RKA) and a budget is allocated for implementation and maintaining HRIS.


3 HMIS PuskesmasDHOPHO

• Information data bank is established. Health facilities are part of a HMIS network.

• One gate system is established.

• Databank is complete and func-tioning.

• Monthly reports are produced.

HMIS is stated in the planning doc of the health facility (RKA) and a budget is allocated for implementation and maintaining HMIS


4 Referral System Pustu, Polindes Puskesmas,Hospitals DHOPHO

• Guideline is avail-able.

• Referral and coun-ter referral letters at Hospital and Puskesmas can be shown.

Availability of records and reports on referral and counter referral of patients. DHO and PHO pro-mote the use.

Planning docu-ments refer to the implementation of the referral guideline, budget is allocated to use it and print the formats.

Documentation of change, decree, local laws

10


No Product User:level of health facility

Availability or pres-ence of product in the health facility (level 1)

Use and implementa-tion of the product(level 2)

Incorporation into Planning and budget documents(level 3)

Action or change induced by the product(level 4)

5 IHPB Puskesmas DHOPHO

IHPB guideline is available. Planning document (RUK) at Puskesmas, and the Integrated work plans (Renja Terpadu) at district & province are available.

Meetings of IHBP process are taking place. The RUK of puskesmas is ac-comodated into the district Renja, the results of Renja is put into RKA of P/DHO.

Planning documents of P/DHO (RKA) men-tion the IHBP process as planning instru-ment and a budget is allocated for the implementation.


6 Integrated Monev

DHOPHO

• Guideline is avail-able.

• Annual Monev report of P/DHO exists.

Monev is conducted yearly. Annual report is used for next plan-ning cycle. Achieve-ments of previous year are compared and analyzed. Results and recommendations are followed up.

Planning doc mention the M&E process and a budget is allocated.


7 P/DHA DHOPHO

• Guidelines are available.

• Availability of yearly P/DHA data.

Data is complete and results are used for management (Plan-ning and Monev) and for advocacy.

Planning documents of P/DHO (RKA) men-tion the development of P/DHA and a budget is allocated for the implementa-tion.


8 Desa Siaga Village PuskesmasDHO

The 5 alert networks are present at village level (notification system, community fund, Transportation system, Blood dona-tion system, Family planning Post).

Recordings and Minutes of Meetings (MoM) on the use of the alert networks (Especially Recording on maternal death at village/sub-village level).

Establishing and maintaining the func-tion of Desa Siaga is mentioned in plan-ning documents and budget is allocated for it.


10 QI-A Hospital • QI team exists.• Application of QI

activities.

• Documentation of the results QI ac-tion. SK that justi-fies existence of QI team. MoM.

Establishing and maintaining of the function of a QI team is mentioned in plan-ning documents and budget is allocated for it.


3.2. Sample

The survey sample was representative for the entire NTB Province. The unit of analysis was the “health

facility” which enabled to answer the indicator. All health facilities have been inventoried for the province

and per district.

See the table on the following page.

11


Table 2. List of health facilities by type and number

No Provinsi/Ka-bupaten/kota

Types and number of health facilities by district Number by Village

Provincial Hospital

PHO District Hos-pital

DHO Puskesmas

1 Provinsi 1 1

2 Kota Mataram

0 1 8 50

3 Lombok Barat 1 1 15 88

4 Lombok Utara 0 1 5 33

5 Lombok Tengah 1 1 24 139

6 Lombok Timur 1 1 29 119

7 Sumbawa Barat

0 1 6 49

8 Sumbawa 1 1 21 164

9 Dompu 1 1 9 63

10 Bima 1 1 20 168

11 Kota Bima 0 1 5 38

Total 1 1 6 10 142 911

The “cluster sampling” technique was chosen for efficiency reasons. NTB was divided into four regions,

namely: Lombok-1, Lombok-2, Sumbawa and Bima. In each of the regions health facilities were selected

using an ad random sampling technique taking into account the stratification for urban and rural facilities.

The stratification for urban and rural health facilities, defined as “distance to the district health office” was

used to ensure fair representation in the sample as these subgroups had maybe significant differences

in results.

At provincial and district level, the public provincial hospital, all district hospitals, the PHO and all DHOs

were automatically included in the sample (100 %). There were 4 districts without district hospital:

Mataram City, North Lombok, West Sumbawa and Bima City.

At subdistrict and village level, a random sampling technique was used to sample Puskesmas and

villages after stratification for geographical distance from the district’s capital city. In the districts where

the number of Puskesmas was considered limited (e.g. North Lombok and West Sumbawa district only

had 5 Puskesmas) all Puskesmas were selected. At village level, three health facilities (Pustu/Polindes/

Poskesdes) were selected adjacent to the selected Puskesmas and based upon the distance to the

Puskesmas (near, middle, far). This resulted at sub-district level in the selection of 63 Puskesmas out

of the total of 142 Puskesmas (44 %) and at village level in a selection of 189 villages out of a total of

911 villages (21 %).

12


The total sample for this survey consists of 459 health facilities (see Table 3).

Table 3. Sample of health facilities by type and number

No Provinsi/Kabupaten/

kota

Jenis dan jumlah fasilitas kesehatan

VillageProvincial Hospital

PHO District Hospital

Puskes-mas

Polindes PosyanduPoskesdes

1 Provinsi 1 1

2 Kota Mataram

1 5 15 15

3 Lombok Barat

1 6 18 18

4 Lombok Utara

1 5 15 15

5 Lombok Tengah

1 8 24 24

6 Lombok Timur

1 10 30 30

7 Sumbawa Barat

1 5 15 15

8 Sumbawa 1 7 21 21

9 Dompu 1 5 15 15

10 Bima 1 7 21 21

11 Kota Bima 1 5 15 15

Total 1 1 10 63 189 189

Grand Total n=459

In a second stage, the unit of analysis was the ”product” to enable the provision of specific information on

the use and implementation of each product and the reasons why it has been used or not. These results

led to the formulation of lessons learnt and recommendations. Not all health facilities applied all products.

Some products are only being used at provincial or at village level. Over the 459 health facilities together

the nine products have been assessed 776 times, as shown in Table 4 on the next page.

Confidence intervals were calculated for the main results. The range of the confidence interval was

defined by the sample statistic + margin of error and the uncertainty was denoted by the confidence

level (95%). The margin of error was the critical value times the standard error, which was the Standard

Deviation divided by the Square Root of the sample size.

13


Table 4. Overview of the Number of surveys per Product and per type of Health facility

Health Facility

ProductsHospital PHO DHO Puskes-

masPolindes Village Total

WISN 7 63 70

HRIS 7 1 10 18

HMIS 1 10 63 74

Referral System 7 63 189 259

IHPB 1 10 63 74

MONEV 1 10 11

P/DHA 1 10 11

Desa Siaga 63 189 252

QI-A 7 7

Total 28 5 50 315 189 189 776

3.3. Survey Team

The Terms of reference (TOR) were developed by the GIZ team and served as base for the development

of a concept paper, a proposal of sampling, methodology and budget by a local research team linked

to an university in NTB (UNRAM). The same team was contracted for both surveys (BLS and FUS) and

consisted out of one principal researcher, four researchers for the district and provincial health facilities

and a team of 20 enumerators for the health centers and villages. Four groups were formed, one for

each region in NTB. Each group had one researcher acting as supervisor and enumerator for the district

and provincial hospital, DHO and PHO. Each group had as well one enumerator functioning as field

coordinator and responsible for data entry. A training was held before each survey to familiarize them

with and refresh their knowledge of the nine products.

3.4. Data Sources and Data Collection Instruments

The surveys were conducted at facility level. The data was obtained from interviews with key informants

in the facilities. Their perception and opinion served as primary and subjective data sources. Only key

informants were interviewed to ensure that as much information as possible about the products was

retrieved. No feedback from patients or other staff members was asked. Their awareness, knowledge

and perception of the products have not been assessed. Calls and revisits were made up to 3 times

in case key informants were absent at the time of the survey. The observations by the survey team in

14


the health facility and from documents such as the annual provincial and district planning and budget

documents served as secondary and objective data sources and were considered as evidence to

confirm the interviews.

Questionnaires for each product for a specific facility were developed as the use of each product

varied according to the health facility level (implementation versus coordination and supervision). The

questionnaires consisted of closed and open questions, respectively for quantitative and qualitative

measurement. For the closed questions multiple options reflecting the degree or scale of achievement

were created. The open questions served to document concrete examples, failure and success factors

for the use of the nine products and for feedback to GIZ.

3.5. Data Entry and Management

A data entry software program was developed, using CS-Pro v4.03, allowing the field coordinators to

enter the data in the field during the survey after checking the completeness of the questionnaires. Data

cleaning and validation was done after completion of data entry. This data was archived with a code

book.

3.6. Data Analysis

For the quantitative data analysis SPSS v15 was used. As the survey used a single-faceted framework,

simple statistical methods such as univariate/bivariate analysis were applied and single and composite

index were computed.

The results were presented in the form of percentages (proportions) and scores. For each result a

subjective value (based on the answers of the respondents) and an objective value (based on observation

and evidence) was calculated. The results were disaggregated for each product, district, level of health

facility and for the BLS for urban and rural facilities. No weighting factors were used.

The results were presented in percentages to answer the indicator “A representative number of health

facilities (at least 80%) in all 10 districts in NTB confirm that they implement or use the guidelines

supported by the project”. These results were based on the existence of the product in the health

facility and reflected “yes or no” questions but they did not bring the extent and quality of the use of the

products into account.

To consider the extent and quality of the use of the products a scoring system has been developed.

15


The score for a product has been based on the four levels of use and the scales/degree of answers for

each of the closed questions (3 to 4 gradations possible). A maximum score of 100 meant that a product

was fully implemented as the guidelines and standards of the projects require, that all conditions were

fulfilled, that all levels of use were fully accomplished (including integration into planning and budgeting

documents and leading to policy change). A score of 100 has been equal to the ideal perfect situation.

All scores together of all nine products defined the overall score.

For the qualitative analysis, factors that influenced the implementation of a products were collected

and grouped under “success factors” if they had led to successful use and under “failure factors” if

they prevented the implementation of the products. An overview of the different types of influencing

factors has been computed and concrete examples for each of the tools have been collected as it was

part of each interview. Concrete examples of change and influence on existing policies as given by the

respondents and observed by the survey team were documented. The FUS focused more in depth in

collecting these qualitative data in comparison with BLS.

3.7. Follow-up Survey (FUS)

The same survey team was hired to conduct FUS. Teams were mixed so that enumerators surveyed other

health facilities than during BLS. A refreshment training and planning session took place beforehand.

The survey sample contained the same facilities and if possible the same respondents. The data

collection instruments were adapted to measure the period after May 2010. More focus was put on the

qualitative part to collect concrete examples; on the “why?” of use or no use of products and the change

the products might had provoked. Technical advisors from the GIZ PAF team accompanied the survey

team during the first 3 days to serve as resource person and to ensure the survey team was obtaining

the required information. The data entry, management and analysis were done in the same way as BLS.

Results of FUS have been compared with the results of the BLS to measure differences which might

had occurred over time. The statistical tests used for this were t-Test for the comparison of means and

the Chi-square test for nominal variables (categorical data) for the comparison of proportions.

16


4. Results

Availability of the products in the health facilities (percentages)

The indicator of the PAF project sets a target that “At least 80 % of health facilities in all 10 districts in

NTB confirm that they implement and use the guidelines supported by the project and this by the end

of the project”.

BLS

The findings of the BLS showed with a 95% confidence that 55.8 ± 3.6 % of all 459 health facilities

confirmed to implement and use the products (subjective value). Evidence that confirmed the “yes/no

questions” on the presence of the product in the health facility led to 52.9 ± 3.4 % of all health facilities

(objective value).

FUS

The findings of the FUS showed with a 95% confidence that 63.4 ± 3.7% or 291 out of all 459 health

facilities confirmed to implement and use the products (subjective value). Evidence that confirmed the

“yes/no questions” on the presence of the products in the health facility led to 255 health facilities or

55.6% ± 3.4% of all health facilities which verifiably used the products (objective value).

Comparison BLS-FUS

Based on the respondents perception (subjective values) there was a significant increase from 55.8 to

63.4 % (p=0.003) but this could not be confirmed with evidence gathered by the survey team. Based

on the objective values, there was no significant difference between the BLS and FUS for the use the

products by the health facilities (p=0.015).

Extent and quality of the use of the products (scores)

To reflect the degree and quality of the use of the products a scoring system has been developed based

on the four levels of use: the availability of the product in the health facility; the implementation and the

fulfilling of all required conditions enabling perfect use of the product; the integration of the products

into planning and budgeting documents; and the influence of the products on planning processes and

policies which might have led to change or action. The scores gave detailed information on each of the

products, the health facility level, the districts and the difference between them.

Note: A score of 100 meant that all products were perfectly implemented and used, planned and

17


budgeted for, and that they produced information and results which led to change or action.

BLS

The overall score for all products together was 34.6 ± 2.6 subjectively (respondents perception) versus

25.8 ± 2.5 objectively (confirmed by evidence) and this with a 95% confidence.

FUS

The overall score for all products together was 34.1 ± 2.7 subjectively versus 26.4 ± 2.4 objectively and

this with a 95% confidence.

Comparison BLS-FUS

There are no significant differences between BLS and FUS for the subjective (t-Test=0,453 and p=0,684)

and objective (t-Test=0,510 and p=0,613) scores.

4.1. Breakdown per Product and Level of Health Facility

Health facilities have different functions, scopes of work, responsibilities and authority at different levels.

Since this is also the case in regards to the products, the health facilities should not be generalized or

treated homogeneously. Therefore a breakdown of the results has been given per level of health facility.

BLS

The results of the BLS in terms of availability and presence of the products in the health facilities are

presented in Table 5 on the following page.

PHO

The PHO used all five products namely HRIS, HMIS, IHPB, Integrated Monev and PHA (100 %).

DHO

All DHOs used HRIS and IHPB (100 %), 8 out of 10 DHOs implemented HMIS and DHA (80 %), and

none of the DHO implemented integrated Monev (0 %).

Hospitals

All Hospitals confirmed to use HRIS (100 %), 86 % implement QI-Action, 57 % use WISN, and none

used the Referral System.

18


Health Center

All Puskesmas claimed to implement Desa Siaga (100 %) but it could only be confirmed in 84 %

(objective value) of the Puskesmas. A similar finding emerged for IHPB, where 73 % of the Puskesmas

respondents confirmed the use but the evidence was only in 44 % (objectively) of the Puskesmas

available. 65 % of the Puskesmas implemented HMIS versus 62 % objectively, 8 % used WISN, and

none of the Puskesmas implemented the Referral System.

Village

At village level, the result showed that none of Polindes/Pustu implemented the Referral System

Guideline and that up to 91% of all villages had evidence for the use of Desa Siaga.

Taking the quality and degree of use into account at each level of health facility, the use of the products,

expressed in scores, was the highest at DHO level with 35 objectively followed by PHO with a score of

32 objectively. The scores for implementation at hospital and village level were the lowest. Table 6 on

the following page shows all details.

Products PHO DHO Hospital Puskesmas Village %

Polindes/Pustu

Village Govern-ment

WISN S 57 8 13

O 57 8 13

HRIS S 100 100 100 100

O 100 100 100 100

HMIS S 100 80 65 68

O 100 80 62 65

Referral System

S 0 0 0 27

O 0 0 0 0

IHPB S 100 100 73 77

O 100 100 44 53

MONEV S 100 0 9

O 100 0 9

P/DHA S 100 80 82

O 100 80 82

Desa Siaga S 100 93 94

O 84 91 90

QI - A S 86 86

O 86 86

Table 5. Percentage of Health Facilities using the products (BLS)

19


Table 6. Overview of the Scores for each Product per level of Health Facility (BLS)

No. Health Facility WISN HRIS HMIS Referral System

IHPB MONEV P/DHA DesaSiaga

QI-A TOTAL

SUBJECTIVE SCORES (Respondent’s perception)

1 PHO 56 30 67 33 17 40

2 Hospital 11 51 0 46 27

3 DHO 52 41 75 0 50 43

4 Puskesmas 3 34 0 38 68 29

5 Village 0 54 27

TOTAL 7 53 35 0 60 17 33 51 46 34,6

OBJECTIVE SCORES (evidence based)

1 PHO 50 15 44 33 17 32

2 Hospital 10 35 0 29 18

3 DHO 41 39 64 0 32 35

4 Puskesmas 1 27 0 26 44 20

5 Village 0 44 22

TOTAL 5 42 27 0 45 17 24 44 29 25.8

Products PHO DHO Hospital Puskesmas Village %

Polindes/Pustu

Village Govern-ment

WISN S 43 27 29

O 43 22 24

HRIS S 100 100 100 100

O 100 100 100 100

HMIS S 100 80 79 81

O 100 80 75 76

Referral System

S 14 8 13 12

O 0 8 3 0

IHPB S 100 100 68 65

O 100 70 52 55

MONEV S 100 70 64

O 100 60 91

P/DHA S 100 90 91

O 100 90 91

Desa Siaga S 97 99 99

O 57 97 87

QI - A S 71 71

O 71 71

Table 7. Overview of the Scores for each Product per level of Health Facility (BLS)

20


FUS

The results of the FUS in terms of availability and presence of the products in the health facilities are

presented in Table 7.

PHO

All five products were used at PHO, namely HRIS, HMIS, IHPB, Integrated Monev and PHA (100 %).

DHO

All DHO used HRIS (100%), 90% used P/DHA, 80 % HMIS, 70% IHPB and 60 % applied Integrated

Monev.

Hospital

All hospitals used HRIS, 71% of the hospitals implemented Quality Improvement Actions and 43% used

WISN. None of the hospitals used the Referral System.

Health Center

The use of products ranged between 75% for HMIS (47 out of 63) and 8% for the use of the Referral

System.

Village

Three % of the surveyed polindes implemented the Referral System. 97% of all villages used Desa

Siaga.

Taking into account the quality and degree of the use of the products, table 8 on the following page gives

a detailed overview of the scores of each tool at each level of health facility. The score was the highest

for DHO with 36 objectively followed by the score for Health Center with 26 objectively.

21


No. Health Facility WISN HRIS HMIS Referral System


QI-A TOTAL

SUBJECTIVE SCORES (Respondent’s perception)

1 PHO 56 15 0 44 17 26

2 Hospital 5 66 5 45 30

3 DHO 53 44 50 34 46 45

4 Puskesmas 17 37 7 38 62 32

5 Village 4 50 27

TOTAL 11 58 32 5 29 39 31 56 45 34.1

OBJECTIVE SCORES (evidence based)

1 PHO 40 15 0 33 17 21

2 Hospital 5 37 2 36 20

3 DHO 46 41 45 6.6 42 36

4 Puskesmas 13 35 6 26 48 26

5 Village 1 44 23

TOTAL 9 41 30 3 24 20 29 46 36 26.4

Table 8. Overview of the Scores for each Product per level of Health Facility (FUS)

Comparison BLS-FUS

Table 9, on the following page, shows that PHO, DHO and villages did not have significant differences

in scores between the BLS and FUS. Their overall score for the use of all products remained stable

between both surveys. But, health centers scored better during the FUS: The overall score for all products

together increased here significantly from 20 to 26 objectively (p=0.007). However, for hospitals the

score decreased significantly during the FUS from 32 to 20 (p=0.006).

The products in detail

Figure 1 shows the scores for each of the products as observed during the BLS and FUS.

BLS

IHPB, Desa Siaga and HRIS were the products with the highest scores, their objective scores were

above 40. They were followed by QI Action, HMIS and P/DHA with objective scores between 20 and 40.

Monev and WISN had very low scores with objective scores below 20. Referral System had a score of

zero.

FUS

Desa Siaga and HRIS continued to be the products with the highest score, their objective scores were

above 40. They were followed by QI-A, HMIS, P/DHA, IHBP and Monev with objective scores between

20 and 40. The lowest scores were for WISN and Referral System and were below 10.

22


0

10

20

30

40

50

60

Table 9: Scores for each Product for BLS and FUS (n=776)

BLS SBLS OFUS SFUS O

WISN HRIS HMIS Referral IHPB MONEV P/DHA Desa Siaga QI-A System

WISN

Questions to calculate the score of WISN included the application of the WISN process and tool in

hospitals and Puskesmas, the production of a report with the results, the use of the report to inform

decision makers, proof of change and action due to WISN. The findings of the BLS showed that the score

for implementation of WISN was 7 subjectively versus 5 objectively. This means that WISN was poorly

applied and in only a few facilities. Based on the feedback from the users, this poor score was mainly

attributable to the lack of knowledge in applying this product as stated by 20 out of 63 respondents in the

Puskesmas. Other constraints were the lack of socialization and training, no supporting budget and staff

rotation. As decisions on deployment of health staff at Puskesmas were taken at a higher level (DHO

en local goverment), the Puskesmas and hospitals did not see the need to apply WISN as the results

produced at their level wouldn’t be taken into consideration by the decision makers. These constraints

were given by up to 17 respondents. In addition, the lack of a binding policy or regulation to implement

WISN (no official endorsement or obligation) prevented staff of computing WISN. Therefore, staff simply

continued to use the existing model based on the use of ratio’s to calculate health human resources

needs and for planning purposes instead of applying WISN.

23


The findings of the FUS showed that the score for implementation of WISN increased to 11 subjectively

versus 9 objectively. WISN was being implemented in 3 out of the 7 hospitals and in 14 out of the 63

Health Centers. In comparison with the BLS this meant a significant increase (p=0.018, see Table 9).

The same reasons were given to explain the low use of WISN.

HRIS

The use of HRIS covered questions on whether HRIS runned properly and data were available

(including software, computer and database), whether databases were maintained and updated

(including anti-virus scans) and the extent to which reports were produced and information was used

for staff development and decision makers. The score for the use of HRIS was 53 subjectively versus

42 objectively for the BLS. The scores were higher at PHO and DHO in comparison with the hospital.

The perceived benefits of HRIS were the easy access to and management of HR information. The main

constraints for the optimal use of this product were related to the incompleteness of individual data

(staff was not obliged to submit data) and the lack of institutionalization of HRIS. The existing system, a

manual excel based system, was still used in parallel and was the preferred system when reports had

to be submitted.

The findings of the FUS showed a score for implementation of HRIS of 58 subjectively versus 41

objectively. Even though HRIS was being implemented at PHO, in all DHO and in all hospitals, the

quality of use remained of the same level as the BLS (p=0.692, see Table 9). The existence of parallel

manual excel based system was noticed during the FUS as well. Problems in the software to produce

valid reports for the management and decision makers inhibited the replacement of the old system by

the new HRIS system. Even though the staff of the Provincial hospital clearly experienced the benefit

of the new system, they could not abandon the old system and confirmed the existence of above

mentioned problems.

HMIS

The perfect use of HMIS included the availability of hardware and software, a functioning HMIS

team (SIKDA team), a LAN/data network, a complete regular updated database, monthly reports, an

established one gate system etc. The score for the implementation of HMIS was 35 subjectively versus

27 objectively during the BLS. HMIS scored the best at DHO level followed by health center level. The

score for HMIS at PHO level was the lowest. The main identified constraints were the absence of an

established SIKDA team, problems with the LAN network, a limited number of computers (stated by

18 respondents), malfunctioning software, viruses, lack of incentives for data entry staff and lack of

stakeholders’ commitment. However, HMIS was widely acknowledged to bring benefits and efficiency

in the daily work: 41 respondents mentioned the improvement of the data quality and completeness,

24


the increase of knowledge, the possibility of having quickly data and reports available. 17 respondents

stated that HMIS has led to more effective and efficient services and 10 respondents answered that

planning did become easier, more accurate and punctual.

The findings of the FUS showed that the score for implementation of HMIS was 32 subjectively versus

30 objectively. This was a small but significant increase in comparison with the findings of the BLS

(0.018, see Table 9). DHO and PHO respondents summarized the reasons for the low score as follows:

the absence of basic conditions e.g. a establishment of a specific HMIS unit in the DHO, lack of anti-

virus programs in the computers, electricity interruption, unfinished software (6 respondents) and the

mismatch between DHO and PHO software (5 respondents). The PHO respondent also mentioned

the additional workload to use the HMIS system. 110 respondents in health centers gave the following

explanations of why the use of HMIS was hampered: lack of specific HMIS operators (29 respondents),

frequent electricity cuts (18 respondents), lack of sufficient budget (18 respondents) and lack of sufficient

and functioning hardware (29 respondents). Positive feedback and comments collected in the health

centers were: HMIS was easy and very practical to use (33 respondents); HMIS produced information

in a very easy and efficient way (28 respondents); HMIS accelerated the delivery of health services (21

respondents); HMIS improved the performance and evaluation activities of programs (12 respondents).

Four health centers claimed they had no problem at all to use HMIS.

Referral System

This product was piloted in West Lombok during 2009. Commitment was obtained by PHO and DHO’s

to implement the guidelines and formats from 2010 onwards. BLS could not retrace the existence of the

product (availability of the new guideline, the formats etc). Only the old government referral system was

used in the health facilities, not the project developed system. Therefore a score of zero was given for

BLS. Reasons for this score were the late start of the activities under the former SISKES project (lack

of time) with lack of commitment and socialization of the product. The question could be raised whether

the new referral system was perceived as being significant different and better than the existing referral

system.

Activities during the Consolidation Program (PAF) to review the Referral guideline and dissemination

of the formats led to slightly better results during FUS. The score for implementation of referral System

was no longer zero but increased to 5 subjectively versus 3 objectively (p=0.006, see Table 9). This low

score was due to the limited number of health facilities using the new referral system: Only 1 out of the

7 hospitals, 5 out of 63 health centers and 6 Polindes out of 189. The quality and degree of the use was

still very poor as different health facility levels were not linked to each other and this condition needed to

be fulfilled to obtain a high score. Hospitals stated that using the new system was not obligatory yet and

25


the lack of support was perceived as an obstacle for successful use. The referral/back referral letters

were found at the provincial hospital covered under dust in a corner of an administrative unit. Positive

feedback from the health centers on the new referral system was received from 12 respondents and

described as no problem in applying the new system, easy data recording, practical formats to be filled

in. 14 respondents gave explanations on why referral scored so low and named the refusal of patients

to be referred, not being used to ask for back referral letters, the administration burden of filling in

the referral letters, and the lack of printed formats. At villages level benefits linked to the new referral

systems were counted 47 times in terms of leading to better services (29 times), a tidier administration,

and a more comprehensive system. Obstacles for a functioning system were transportation problems

(10 respondents), the low awareness of the community to ask for/bring the letters and to be referred (15

respondents), the costs of referral and the administrative burden.

IHPB

Implementation of IHPB was verified in terms of availability of IHPB guidelines, their use in the planning

process in 2007-2009 (BLS) and in 2010-2011 (FUS), and the extent to which the Puskesmas and

DHO’s planning was accommodated into respectively the DHO and PHO’s planning. Implementation of

IHPB scored 60 subjectively versus 45 objectively during the BLS. IHBP had the highest scores at DHO

level followed by PHO level. The scores were the lowest at Puskesmas level. The benefits of IHPB in

terms of more transparency in planning, less overlap and more synergy, better coordination and easier

planning development were confirmed 87 times.

The main constraint for the use of IHBP at Puskesmas level was the conflict IHBP created being a

bottom-up process while in reality planning and budgeting processes still take place top-down. This was

stated by 20 respondents. Other reported constraints included lack of understanding and facilitation of

the process, and limited resources and budget to support the implementation.

The findings of FUS showed scores dropping significantly from 60 to 30 subjectively and from 45 to

24 objectively (p=0.001, see Table 9). This was the only product with a significant decrease in score in

comparison with the BLS. The main reason for this score was the obligatory use of P2KT or the MoH

version for Integrated Health Planning and budgeting which was different from the project developed

IHBP.

Integrated Monev

The assessment of the use of Monev at PHO and DHO included questions on the conduct of Integrated

Monev in 2008, the follow up of the recommendations in 2009 for BLS and in 2010 for FUS, and the

extent to which the results were used as input in planning processes. The implementation of Monev

scored 17 subjectively and objectively during BLS. While Monev for each program ran regularly, the

conduct of Integrated Monev was constrained by the non compliance to the schedule by the different

26


programs. The score at DHO level was zero as it was not conducted at district level. The absence of

budget for supporting Integrated Monev was predominantly stated by DHO’s respondents to explain

why. The benefits of Integrated Monev were identified as bringing improving effective and efficient

program implementation, identifying DHO performance more comprehensively and offering a tool to

find solutions for expressed problems. But these benefits were not convincing enough for the DHOs to

conduct Integrated Monev. The findings of FUS showed that the implementation of Integrated Monev

scored 39 subjectively and 20 objectively but without significant difference with the BLS results (p=0.180,

see Table 9). Since BLS, Integrated Monev was conducted once and the perceived benefits, stated by 5

DHO, were that solutions and feedback were given right away during the Integrated Monev mission and

that the result based approach was very useful. The PHO repeated how difficult it was to coordinate a

joint mission which matched the agenda of each program.

P/DHA

The implementation of P/DHA included questions on the availability of P/DHA for 2007-2008 (BLS)

and 2009 (FUS), the completeness of data, and the use of the data as input for the IHPB process and

advocacy. The score for P/DHA was 33 subjectively versus 24 objectively during BLS. P/DHA were

elaborated for 2007 and 2008 but not for 2009. The scores at DHO were higher than at PHO level.

Reasons for the low scores were the unavailability and incompleteness of data, the mismatch between

available data and the templates as reformatting and adjustments, required before data entry, were not

done. Lastly, producing P/DHA was not compulsory so in absence of a request, facilitation and technical

backstopping by the project, the production of P/DHA reports did not continue. Budget constraints to

fund all the work meetings were mentioned as well at DHO level which pointed out the dependence of

development agency’s support once more. Perceived benefits of conducting P/DHA were the increased

understanding of fund sources and budget allocation, and the increased availability of information that

could be used for advocacy and evaluation.

FUS findings showed scores for P/DHA of 31 subjectively versus 29 objectively which were slightly

better than the scores of BLS (p=0.003, see Table 9). PHO and 9 out of 10 DHO applied P/DHA but

the quality of implementation was weak. PHO stated that there was a difference between the tool

promoted by the project and perceived as rather complicated, and the tool promoted by the University

of Indonesia and more widely used. Feedback of the DHO was the incompleteness of required data (7

respondents) and complexity of the tool (4 respondents). As positive feedback DHO stated that DHA

produced data which can be used for advocacy to make budget allocation fairer according to priorities

(11 respondents).

27


Desa Siaga

The score for the implementation of Desa Siaga referred to the availability (establishment) of the

alert network systems in the village and to the extent these functions were maintained. Desa Siaga

implementation scored 61 subjectively and 44 objectively for BLS. The success of Desa Siaga was due

to the perceived benefits by the villagers: increased solidarity and awareness in the community, better

access to information and quicker service delivery when needed. The constraints identified for the

implementation were the lack of coordination and facilitation by DHO and the departure and drop-out

of village facilitators.

FUS revealed for Desa Siaga implementation scores of 56 subjectively and 46 objectively. The increase

of the objective scores was significant (p=0.028, see Table 9). Perceived benefits and reasons why

Desa Siaga was working well were formulated by 92 respondents in terms of better services for the

community, increased community awareness on health problems and improved health status of pregnant

women and children. Feedback from the villages why Desa Siaga was not working well included the

lack of supporting budget and support of health facilities (21 respondents), lack of coordination (8

respondents); lack of socialization of the concept and unawareness of the community members (18

respondents). 18 respondents stated that Desa Siaga was not running in their village.

QI-Action

The QI-Action covered questions on the occurrence of QI-Activities in hospital units, the existence of a

functioning QI team, proof of change, and the existence of a plan and budget to implement QI-Action

in 2010 for BLS and in 2011 for FUS. The findings of BLS showed a subjective score of 46 versus 29

objectively. This difference is due to the absence of real evidence. Budget constraints and staff rotation

were stated by the respondents in the hospital as main reason for the low score. The findings of the FUS

showed that QI-Action was implemented in 5 out of 7 hospitals and that the score for implementation

of QI-Action was 45 subjectively versus 36 objectively. There was no significant difference with the BLS

(p=0.289, see Table 9). The same reasons as during BLS were stated as reason for the low use. In

addition the lack of commitment of decision makers and the missing legal framework to make QI-Action

obligatory in the hospitals were mentioned by the hospital staff.

Comparison BLS-FUS

Table 9 shows the differences between the BLS and FUS. For the overall score there was no significant

difference between the results of the BLS and FUS. However, when breaking down the results per

product, significant differences were found: Four products scored better, namely WISN, Referral System,

P/DHA and Desa Siaga but with minimal improvements of scores. Only one product scored significantly

worse. The score of IHPB dropped to half the value in the period between the BLS and FUS. The reason

28


given was the mandatory use of the P2KT guideline. No difference in status for three products, namely

HRIS, Integrated Monev and QI-Action.

Table 9. Comparison scores BLS-FUS for each product and level of health facility.

HealthFacility

WISN HRIS HMIS ReferralSystem


QI-A Total t-Testp-value

Objective Scores (evidence based) – BLS versus FUS

PHO BLS 35 15 44 33 17 18 t=1.697 p=0.103

FUS 40 15 0 33 17 21

Hospital BLS 10 50 0 29 32 t=2.950p=0.006

FUS 45 37 2 36 20

DHO BLS 41 39 64 0 32 35 t=0.95p=0.349

FUS 46 41 45 7 42 36

HealthCenter

BLS 1 27 0 26 44 20 t=2.885 p=0.007

FUS 13 35 6 26 48 26

Village BLS 0 44 22 t=1.238p=0.232

FUS 1 44 23

TOTAL BLS 5 42 27 0 49 17 24 44 29 25.826.4

FUS 9 41 30 3 24 20 29 46 36

t-Testv-value

t=2.556p=0.018

t=0.404 p=0.692

t=2.556p=0.018

t=4.812p=0.006

t=4.483p=0.001

t=1.431p=0.180

t=5.477p=0.003

t=3.070p=0.028

t=1.162p=0.289

t=0.510p=0.613

4.2. Breakdown per District and Geographical Distance between health centers/villages and the DHO

Use of the products per district

NTB province has 10 districts and all districts were equally supported in the use of all nine products

except for the Referral System. This was only piloted in West-Lombok district during SISKES project

(See section 4.1).

BLS

Table 10 shows the scores, objectively and subjectively, for each of the districts and for each of the

products. The scores for the products at provincial level were not included in the calculation of this

breakdown. Four districts did not have a district hospital so QI-A could not be assessed there. North

Lombok was a new district since 2009 so P/DHA could not be assessed as no budget data was available

for 2008.

29


No Districts WISN HRIS HMIS Referral System

IHPB Monev P/ DHA Desa Siaga

QI-Action Total

Subjective Scores

1 West Lombok 15 56 28 0 57 0 42 70 50 37

2 North Lombok 0 50 43 0 69 0 42 37

3 Central Lombok 17 53 54 0 66 0 58 62 50 40

4 East Lombok 10 56 43 0 63 0 58 59 50 38

5 West Sumbawa 0 67 41 0 64 0 58 74 38

6 Sumbawa 0 59 38 0 71 0 58 74 50 39

7 Dompu 3 56 67 0 28 0 58 46 25 31

8 Bima 0 50 42 0 36 0 58 72 50 34

9 Bima City 50 0 0 42 0 25 77 24

10 Mataram City 67 9 0 52 0 17 25 21

Total 4 56 37 0 55 0 48 61 28 32

Objective Scores

1 West Lombok 13 44 28 0 53 0 25 47 50 29

2 North Lombok 0 37 31 0 58 0 25 22

3 Central Lombok 8 53 51 0 62 0 50 57 13 33

4 East Lombok 9 40 40 0 49 0 50 57 13 29

5 West Sumbawa 0 48 36 0 64 0 50 58 32

6 Sumbawa 0 56 38 0 62 0 25 63 50 33

7 Dompu 0 51 54 0 28 0 50 17 25 25

8 Bima 0 35 35 0 25 0 50 36 25 23

9 Bima City 0 15 0 0 11 0 0 31 7

10 Mataram City 0 56 7 0 28 0 17 19 16

Total 3 43 32 0 44 0 32 44 18 25

The objective and subjective scores had the same ranking for the districts. The differences between the

objective and subjective scores were minimal except for North Lombok where the subjective score of

37 was much higher than the confirmed objective score of 22. The best scoring districts were Central

Lombok and Sumbawa with an overall objective score of 33, followed by West Sumbawa, West Lombok

and East Lombok. The districts with the lowest score were Mataram City and Bima city with respectively

16 and 7 objectively. All products had low scores in these two districts except for HRIS in Mataram

City and Desa Siaga in Bima City. The survey could not give reasons for this difference between “city

districts” and “larger districts” as further Focus Group Discussions and in depth interviews would be

needed and this was beyond the scope of this survey.

Table 10. Breakdown of the scores for each product and district (BLS).

30


The products with the highest scores at district level were IHBP, HRIS and Desa Siaga. IHBP and Desa

Siaga were best applied in West Sumbawa and HRIS is Mataram City (the district that in general scored

low for all products together). Referral System and Integrated Monev were zero in all districts. WISN was

only applied in 4 districts subjectively and confirmed in 3 districts but the extent of use was very limited

(maximum objective score of 13 in West Lombok).

FUS

Table 11 shows the scores, objectively and subjectively, for each of the districts and for each of the

products for FUS.

No Districts WISN HRIS HMIS Referral System

IHPB Monev P/ DHA Desa Siaga

QI-Action Total

Subjective Scores

1 West Lombok 25 61 43 54 58 63 38 63 50 50

2 North Lombok 0 50 35 0 51 63 38 56 37

3 Central Lombok 27 72 54 0 54 47 56 76 50 49

4 East Lombok 22 54 45 0 42 0 56 75 50 38

5 West Sumbawa 0 66 40 0 56 47 56 78 43

6 Sumbawa 0 76 43 0 53 47 28 70 50 41

7 Dompu 0 70 61 0 40 0 28 36 35 30

8 Bima 0 48 39 0 37 47 56 37 35 33

9 Bima City 0 20 18 0 25 47 0 42 22

10 Mataram City 33 76 27 0 25 0 28 26 24

Total 11 59 40 5 44 36 39 56 27 37

Objective Scores

1 West Lombok 23 42 37 27 37 33 33 53 50 37

2 North Lombok 0 35 30 0 32 33 33 48 23

3 Central Lombok 21 50 46 0 34 25 50 64 50 38

4 East Lombok 18 37 38 0 27 0 50 64 50 32

5 West Sumbawa 0 46 34 0 35 25 50 66 28

6 Sumbawa 0 53 37 0 34 25 25 61 50 32

7 Dompu 0 49 52 0 25 0 25 29 25 23

8 Bima 0 33 34 0 24 25 50 31 25 25

9 Bima City 0 14 15 0 16 25 0 35 12

10 Mataram City 27 53 24 0 16 0 25 22 18

Total 9 41 35 3 28 19 34 47 25 27

Table 11. Breakdown of the scores for each product and district (FUS).

31


The objective and subjective scores had more or less the same ranking for all districts. The subjective

scores were in average 5 to 10 points higher than the objective scores. The biggest difference between

both scores was again for North Lombok with a subjective score of 50 versus the confirmed objective

score of 37.

The best scoring districts were Central Lombok and West Lombok with an objective score of 38 and

37, followed by East Lombok and Sumbawa with a score of 32. The districts with the lowest score were

Mataram City and Bima city with respectively 18 and 12 objectively due to the scores of zero for some

of the products. Reasons why “cities” scored lower than “districts” were not given by the Survey team

as exploring this was beyond the scope of the survey.

The products with the highest scores at district level were HRIS and Desa Siaga. IHBP was no longer

part of the best three products at district level. HRIS was best applied in Sumbawa, Mataram City (the

district that in general scored low for all products together) and Central Lombok. Desa Siaga was best

applied in West Sumbawa, East and Central Lombok. Referral System and Integrated Monev had no

longer zero scores for all districts. Referral System had been applied now in West-Lombok. Integrated

Monev was now applied in seven out of ten districts. WISN continued to be used in only four districts

and the maximum score for implementation was 27.

Stratification for geographical distance

The assumption was that the further away health facilities and villages were located from the DHO,

the lower the scores would be, as it was assumed that DHO through its regulating, coordinating and

supervisory role could influence the implementation of products. The results of the BLS however showed

no statistically significant difference between health centers and villages close by or far away from

DHO. The only exceptions were the use of HMIS data for the IHPB process and the accommodation of

Puskesmas planning into the district planning. Here the survey showed that the scores decreased with

increasing geographical distance from DHO to the health centers (p< 0.050). Therefore this breakdown

had not been repeated during the FUS.

4.3. Integration of the products into planning and budget documents

The integration of the use of the products in planning and budget documents was the third level of use

of the products that was assessed during the survey. Integration of the use of products in planning

documents was considered as a required condition for sustainability. But, as integration into planning

documents did not automatically mean that the activities would be executed and that there would have

been budget allocated for implementation, the survey only assessed budget documents. The results

32


referred to confirmation from the respondents whether budget was allocated for the implementation and

further use of the products and the amount of the budget allocation. A possible bias could have occurred

here as in principle not all products needed budget each year to guarantee their sustainability.

For example for WISN, once all staff members were trained (this included costs for travel, manuals,

training room, resource person etc), doing the exercise of calculating the staffing needs and producing

a report did not need resources except time allocation of staff. But “no budget, so no activity” was stated

as common practice.

In general the surveys pointed out that there was limited budget allocation for the products as such and

that proof of budget allocation was difficult to retrieve. Table 12 shows the percentage of health facilities

during the BLS that had a budget allocated for each of the products. Table 13 shows the same results

for the FUS. Mostly it was integrated into a more general budget line, for example, budget for HRIS

in the provincial hospital was part of the overall information department’s budget. The amounts were

perceived by the respondents as being insufficient for full implementation of the product.

Table 12. Percentages of Health Facilities having Budget in 2010 to support the implementation of Products (BLS)

Products Health Facilities

PHO<n=1>

DHO <n=10>

Hospital <n=7>

Puskesmas <n=63>

Subjective Objective Subjective Objective Subjective Objective Subjective Objective

WISN 0 0 0 0

HRIS 100 0 20 10 0 0

HMIS 0 0 80 80 14 8

Referral System

0 0 0 0

IHPB 100 0 50 40 30 27

MONEV 0 0 30 20

P/DHA 0 0 10 0

DESA SIAGA 44 32

QI-A 29 0

33


Products Health Facilities

PHO<n=1>

DHO <n=10>

Hospital <n=7>

Puskesmas <n=63>

Subjective Objective Subjective Objective Subjective Objective Subjective Objective

WISN 0 0 2 2

HRIS 100 100 20 20 0 0

HMIS 100 100 90 80 52 32

Referral System

14 14 0 0

IHPB 0 0 80 80 60 51

MONEV 100 100 40 40

P/DHA 0 0 50 50

DESA SIAGA 62 56

QI-A 29 29

Table 13. Percentages of Health Facilities having Budget in 2011 to support the implementation of Products (FUS)

PHO

The results of the BLS showed that there was no specific budget allocated for the further use of the

products. Even though PHO respondents thought there was budget allocation for HRIS, the planning

and budget documents of the PHO contained no proof of this. The FUS however found better results as

thorough examination of the budget documents for 2011 showed budget allocation for HRIS, HMIS and

Integrated Monev. There was no budget to support the process of IHPB or to conduct P/DHA.

DHO

The BLS showed that 80% of the DHO had evidence for budget allocation for HMIS and 40 % for

implementation of IHBP. But only 20 % (= two districts) had budget to conduct Integrated Monev. The

allocated budgets were perceived by the respondents as being insufficient. There was no budget

to produce DHA in 2010. FUS showed better results in number of districts that had specific budget

allocation for the use of the products: The number of districts that allocated budget doubled for HRIS,

IHPB and Integrated Monev. Five out of the 10 districts had now budget to execute DHA. Budget for

HMIS remained available in 80% of the districts.

Hospitals

The BLS could not find proof for budget allocation to sustain the use of any of the products in the hospital

budget documents. This was confirmed by the answers of the respondents except for QI-A. 29% of the

respondents were convinced there was budget available for QI-A but this could not be proven. However,

the FUS could retrieve proof for specific budget allocation in 2011 for the implementation of the Referral

34


System and the QI-Action. For HRIS, the provincial hospital had no specific budget line as such but the

budget under “information systems” could be used for HRIS.

Health Center

Results of BLS: Almost one out of three Health Centers had proof for the establishment of Desa Siaga

and implementation of IHBP, respectively 32 and 27%. Only eight percent of the Puskesmas had budget

in 2010 to support HMIS activities. There was no budget allocation for WISN and the Referral System.

FUS could find more evidence for specific budget allocation to continue the use of the products. Three

Health Centers (2%) had now budget for WISN, four times more health centers (32%) had budget for

HMIS applications, the number of PKM with budget for IHBP almost doubled (from 27 to 51%) and more

than half of the health centers had now budget for Desa Siaga (56%).

4.4. Input for planning, action and policy change

The surveys examined whether the information produced by the products was used in planning

processes, had led to policy change, concrete action or change of an existing situation. This assessment

was considered as the fourth level of use.

BLS

• The BLS found little evidence of products producing information which was then used for general

planning processes. There were 29 concrete examples:

• WISN results were used for HR planning in one Puskesmas out of six in West Lombok.

• HRIS data have been used for HR planning and staff development in four out of ten DHO (40%),

namely Mataram City, East Lombok, West Sumbawa and Sumbawa.

• Health information produced by HMIS was used in the planning cycle of two out of 10 DHOs (20%),

namely in DHO Central Lombok and in DHO Dompu and also in several health centers: one

out of six Puskesmas in West Lombok, three out of eight in Lombok Tengah, three out of ten in

East Lombok, two out of five in West Sumbawa, three out of seven in Bima and two out of seven

Puskesmas in Dompu.

• The results of DHA were used in the planning cycles of eight out of 10 DHO (80%), except for

Mataram City and North Lombok.

None of the products has lead to change of an existing policy.

The BLS collected 81 concrete examples of action and change. Change was often formulated as

improvement of existing processes.

35


The information produced by the HRIS made staff promotion possible at DHO Mataram.

DHO Loteng and Dompu used the data from HMIS to make tables and graph for advocacy and

promotion. All Puskesmas respondents, except in Mataram City, confirmed they used the HMIS data

for better input for health promotion activities and advocacy, for better and quicker response in case of

disease outbreak, and to improve health service delivery.

IHPB to improve planning and budgeting processes was felt significantly in 3 out of 10 DHOs (30%) and

only a little in 5 DHOs.

DHA: 8 out of 10 DHOs (80%) were able to show they used the results of DHA for advocacy; 4 out of 8

DHOs confirmed that advocacy had moderate effect on budget allocation, in particular for MNCH.

FUS

Even though during the FUS more focus was put on this part of the survey, the interviews could not

retrieve more concrete facts or evidence for the fourth level of use.

• The FUS found evidence of products producing information which was then used for general

planning processes in 35 concrete cases:

• WISN: Four health facilities (out of 70) used WISN results as input for personnel staffing: the district

hospital of East Lombok as the only hospital and three health centers.

• HRIS: Since June 2010 no evidence could be found for DHO, only Bima hospital had real proof of

data being used for personnel planning.

• HMIS: the DHO of Lombok Barat and North Lombok used information produced by HMIS for planning

processes, 28 health centers out of 63 had proof they used HMIS data for planning processes.

None of the products had lead to change of an existing policy.

The FUS collected 78 concrete examples of action and change. But changes were formulated by the

respondents as improvement of processes and then called “good practice”. There was no proof of real

action.

36


4.5. Qualitative results

For each product the survey team asked for factors that influenced the use of the product and that

were critical for sustainable use. The factors identified by the respondents were general conditions

that needed to be fulfilled and were very specific factors for a particular product in a particular health

facility. Section 4.2 gives for each tool specific influencing factors. In this section an overview of all kind

of factors is given, distinguishing them into two groups, namely the success versus the failure factors.

Failure factors were often the absence or lack of success factors and vice versa. The respondents

formulated the influencing factors mostly in terms of failure factors and less in terms of positive success

factors. Out of 1143 factors collected in the questionnaires of BLS, 491 were formulated as success

factors (43%) and 652 as failure factors (57%). FUS collected 800 factors, of which 374 (47 %) were

identified as success factors and 426 (53%) were identified as failure factors.

Success Factors

The most mentioned factor for successful use of a product was the perceived benefit by the users, and

the contribution of the products to better and more efficient achievement of work objectives. If the product

led to more efficient use of the staff’s time they were willing to invest in it. If a tool was user-friendly

and functioned well e.g. a software program, they were keen to apply it. Commitment and motivation

expressed by higher level officials reinforced the use at a lower level facility. Even though benefits for

each product were widely acknowledged and easily identified by the respondents, if too many failure

factors were present at the same time e.g. lack of legal framework, budget etc the perceived benefits

and commitment would not led to the use of the product.

Failure factors

Obstacles or challenges for the use of products are summarized in the following categories:

• Financial factors: insufficient or no budget allocation for the implementation e.g. for meetings,

infrastructure, supervision visits and trainings. No harmonization between budgets of the different

health facility levels for the use of the same product and lack of synchronization in budget

disbursement. No incentive linked to the use of the product.

• Human Resource (HR) factors: lack of (competent) staff to use the product. Too frequent staff

rotations and retirement of people familiar with the use of the product. No culture of proper handing-

over or maintaining institutional memory. Products were considered as additional workload, staff

had not enough time to use the product. Lack of ownership, no one specifically in charge for the

product. Training: not all staff was trained in the use of the products, content of training was not

37


always consistent with the details of the product, troubleshooting was often not included in the

training.

• Technical factors: insufficient infrastructure to support the use of products: insufficient hardware

(computers, virus scan and server capacity), malfunctioning software, frequent power cuts, limited

internet connection. Application of some products was perceived as complicated and difficult.

Insufficient support by technical guidelines, products not always adapted to the existing formats or

data. Products were not compatible, flexible and responsive to change in case new requirements of

decision makers occurred (HMIS, HRIS and IHPB were perceived as less useful after change of the

official reporting formats and their lack of adaptation to the new requirements).

• Legal factors: absence of a legal basis or framework for implementation of the product and for

collaboration between different levels and departments e.g. obligation and permission to use the

product, and timely and sufficient data sharing. Results produced at lower level facilities (products

with bottom-up approach) could not be used for decision making as decision were taken higher up

using the old existing system. Unclear policies.

• Organizational factors: Lack of coordination, lack of communication and socialization, no willingness

to collaborate between different levels and departments of one level, no ownership.

• Community factors: Low awareness of health problems. Refusal or delay in seeking help. Persistent

local traditions. Remote areas.

• Attitude, cultural factors: lack of attention and interest for data of high quality. Dishonesty in data

provision. No transparency in budgets. Fear for new products and change. Existing system was

considered as better. Need for change or innovation was not perceived.

4.6. Limitations of the Survey

It was a limited health facility survey. Only GIZ products had been assessed. Existing systems which

were not developed by the project e.g. referral system, HMIS etc. had not been investigated. Therefore

the results did not reflect the health system’s performance in terms of service delivery, competence and

quality.

Only key informants had been interviewed to ensure that as much information as possible about the

products could be retrieved. No feedback from patients, other staff members had been asked. Their

38


awareness, knowledge and perception of the products had not been assessed.

The formulation of the indicator was rather imprecise. It stated that “at least 80 % of health facilities in

all 10 districts in NTB confirm that they implement and use the guidelines supported by the project and

this by the end of the project”. It did not precise which guidelines and how many in each of the health

facilities. Therefore arbitrary decisions for the methodology had to be made.

The objective scores were sometimes underestimating the use of the products as proof and evidence

could not been shown at the time of the interview. Reasons and pretexts for the absence of evidence

and oral confirmation were not accepted as proof.

The time interval between the BLS and FUS was only 11 months and thus very short for policy work to

have its effects. However this could not been organized differently due to the project duration of PAF in

NTB (January 2010 - June 2011).

39


5. Interpretation of the results

The former GIZ projects, SISKES and HRD, invested high amount of resources in terms of time, money,

human resources and expertise, into the development of products. Providing tools, manuals, information

systems and products was part of a comprehensive strategic plan to strengthen the district health

systems and the human resource system. During the project duration the perfect conditions were

created for successful and sustainable implementation of the products: The products were designed

and developed in close collaboration with the partner to promote ownership and integration in the

existing system; the products were improvements and adaptations of existing mechanism; the products

were not simple stand alone tools but all products were seen as a means to support and achieve a

higher purpose, namely the provincial Maternal, Neonatal and Child Health (MNCH) strategy “AKINO”

or “no maternal deaths in the village”; workshops were held to gain commitment of high level decision

makers at province and district level to continue the use of the products. The final achievements of

SISKES and HRD projects regarding to use of products and coverage over the province were very

positive. However the BLS and FUS, only five and 16 months after the end of the projects, could not

confirm the sustainable use of the products. The results were considered disappointing but realistic.

Why were the results that low? The survey made sure that only key products were selected, and that all

respondents were key persons familiar with the projects and the products. The main reasons to explain

the low scores were the abundant presence of failure factors, resistance for change, the complexity of

achieving results and sustainability through aid development interventions and the demanding character

of the survey which was looking for perfection in the application of a product (only then a score of 100

could be given).

Too many Failure Factors

Without any doubt the main factor was the absence of a binding legal framework which could

institutionalize the products within the system, obliging the health facilities to use the newly developed

products and to put aside failing or old mechanisms. Without rules and regulations the staff would not

have done things differently. The focus of the Consolidation Program 2010-2011 was to consolidate

what had been started under SISKES and HRD through policy work at provincial and district level,

linked with the central level. But the 1.5 year in NTB had been too short to change the partner’s mindset

from implementation to policy work and to issue local laws (decree, Perdas or Pergubs) to mainstream

40


the products. The policy process from drafting the local laws to approval and dissemination easily takes

up more than 1.5 year. Development agencies at province and district level in NTB work in general on

the implementation level and partners were not used to policy work.

HR constraints were the second important group of factors responsible for the low scores. Trained and

or key staff were rotated frequently and this without notice or proper hand over to colleagues. Workload

was often not properly distributed and the new products increased the workload when the old systems

continued to exist in parallel. A clear example was the change of director in the provincial hospital in

Mataram. The previous director knew both projects very well and facilitated within the hospital the

implementation of strategies and products. He even issued local hospital decrees and circular letters to

reinforce the implementation and approved the creation of a patient safety workgroup. The new director

was not aware of these GIZ introduced products which should have replaced the old systems. As time

goes by the scores will continue to decrease and old systems will take over again.

Financial constraints were other important failure factors. There was limited to no budget allocated for the

use of the products. Implementation, planning or data collection processes, maintenance, work meetings

and trainings could not be done without budget. But no budget could have been made available if the

planning documents were not mentioning these activities. Products that were being used at different

health facility levels needed budget disbursement at these different levels and in a coordinated and

synchronized way to enable the implementation. If the use of the products was made obligatory by law,

the integration of the products into planning and budgeting documents would have been easier.

The survey showed that failure factors were often closely linked to each other (interdependency) and that

too many failure factors present at the same time prevented a product of being fully used. These factors

and the survey results reflect the weak structural and managerial capacities of the health system and the

complexity one had to deal with when introducing new mechanisms, product and tools to improve the

existing system.

Fear for change

That people are afraid for change and prefer to stick to what they are used to is generally known and

accepted. The products however were developed to bring change and to improve the health systems but

little evidence could be collected on how the products achieved this objective. Some of the products e.g.

HMIS, WISN and HRIS, were considered as “Bottom-up” products as they produced relevant information

in the health centers and hospitals which could be used for decision making at DHO and PHO level. But

often this information was ignored and did not find its way up to the policy and decision makers. Decisions

continued to be made based on the use of the old mechanism and the products failed to bring change.

41


Complexity of sustainability

Development agencies often need quick, visible and tangible results. Products, guidelines, books, manual

and trainings are often the preferred way of showing to the donor what has been done. Guaranteeing

sustainability of these products after the project duration is a huge challenge as shown by the surveys.

On top of that new products and tools have the danger of creating duplication, increasing the workload

of staff and adding to the administrative burden. It seems that both projects were not an exception on

these international findings and that lessons can be learnt for future design and interventions (see

section 6).

However, products are developed as tool to reach/achieve certain norms and standards which contribute

to strengthening the health system and improving management and quality of health services. Sustaining

norms and standards needs human and financial resources, planning and M&E efforts. A project can be

used to establish and develop these tools and norms based on international expertise and standards

but the maintenance needs effort of the partners in terms of budget, integration into planning and

budget documents and institutionalization. Policy work is an absolute must.

Looking for perfection

Another reason to explain the results, particularly in regard to the significant differences between the

availability and presence of the products, expressed by the percentages, and the extent or quality of use,

expressed by the scores, was the demanding character of the survey’s methodology. The questions to

calculate the availability of the products in the health facilities needed only a simple “yes/no” answer and

did not asses to what extent the products are being used. The scores however included four levels of

use, a high demand for perfectionism and a strict judgment by the survey team. Each of these questions

had 3 to 4 options as answer enabling a precise reflection of the degree of implementation (fully, mostly,

slightly, and not at all). All conditions and requirements had to be fulfilled before the maximum score was

given and the four levels of use were hierarchical calculated so that the next level never had a higher

score then the level below. The survey team was trained to be strict and rigorous and to look for the

ideal perfect situation which was reflected by a maximum score of 100. Therefore the percentages were

in general much higher than the scores.

This also explained the differences between the subjective and objective values even though these

differences were much smaller. By questioning well informed key respondents familiar with the products

the correct information was gained straight from the start and could be proved accordingly by observation

and consulting documents. As stated above the survey team was strict if evidence was not available

(see section 4.6).

42


The differences in results between the different health facility levels could be explained by the way the

projects worked with each of the facilities. The projects consisted of GIZ and the partner, in the case

of NTB, the PHO and DHO were the direct implementing agencies. Therefore, the direct collaboration

between GIZ staff and staff of DHO could explain the better results in the use of the products at DHO

level. Implementation, replication and roll out of the products at the Health Centers and villages were

under the responsibility of the PHO and DHO and explained the lower scores at these health facilities.

Hospitals fall under the local government. The exception is Desa Siaga as there the project was directly

involved with the implementation and the approach turned out to be a success. FUS showed a significant

decrease at hospital level mainly due to staff rotation and lack of legal framework to mainstream the

products.

Minimal to no differences between BLS and FUS

The overall results were not significant different between BLS and FUS mainly because the time

interval was only 11 months and the technical advisors of GIZ were still active in NTB working on

the institutionalizing of the products through policy work and by providing technical advice for the

implementation of the tools especially for Referral System, HMIS, HRIS and WISN. When the results

were broken down per product and level of health facility significant differences did exist but these were

all minor except for hospitals and IHPB for reasons stated above.

43


6. Lessons Learnt

Based on the results that reflected the current situation and their underlying reasons, a number of

lessons learnt can be formulated. These can serve as recommendations and guidance for future design

of interventions and development of products.

• All influencing factors should be considered and addressed. All conditions and requirements for

successful use of products have to be fulfilled. Legal frameworks, sufficient budget and proper HR

structures have to be in place to mainstream products. Without perceived benefits a product will not

be used.

• A health system with weak institutional and management structures will not facilitate the full

integration and use of a new product and cannot ensure their sustainability.

• Products which provide evidence on a lower health facility level (Bottom-up approach) need

commitment and back up from higher level policy and decision makers; otherwise these products

cannot bring change.

• A period of less than two years is too short to create legal structures for the institutionalization of

products. This has to be built in from the start of product development and design.

• Full ownership has to be created and stimulated from the design onwards. Products developed

during projects have many dangers: they are often agency (donor) driven in the need for quick,

tangible and visible results, they may create duplication and add to the administrative burden and

staff’s workload.

• Shifting mindsets from implementation to policy interventions at decentralized level takes time.

• Even though products serve a higher goal, e.g. AKINO in NTB, it is not a guarantee for sustainable

use.

• One size does not fit all. Newly designed products have to be compatible with existing formats and

systems without requiring too much change or adjustments to fit in. At the same time they have

to be easily changeable and adaptable to new requirements of decision makers or they will be

abandoned.

44


7. Conclusion

Both projects, SISKES and HRD, have put a lot of efforts in the development of useful tools to assist the

health offices and health facilities to improve their management and service delivery system. As long as

these projects were running the implementation coverage and use was impressive. The Consolidation

Program focused on policy efforts to create legal frameworks and the necessary laws and regulations

to support the sustainable use of these products.

Despite all these efforts the level of use could not be maintained and the target of 80% of all health

facilities using the products has not been achieved. Both surveys were able to demonstrate the

availability and presence of the nine tools in the health facilities (BLS 52.9 % and FUS 55.6%). When

quality was brought into account with a scoring system, the results dropped further with 25.8 for the BLS

and 26.4 for the FUS. The products with the highest scores were Desa Siaga, HRIS and IHBP. Use of

the products was the highest at DHO level. Central Lombok and Sumbawa were the two districts with

the highest scores.

The surveys identified multiple factors which influenced successful use. Even though the products

provided valuable data and information and staff perceived the benefits of using the products in their

daily work, the quality of using the products and the degree of incorporation of the products and their

results into policy were limited. Main obstacles, identified by key staff were absence of legal frameworks,

the frequent staff rotation without proper handing over and insufficient budget allocation.

Fear for change and the effect of development agencies with high amounts of funding, staff and expertise

during a short period of time were factors that influenced the sustainability of newly developed products.

45


8. Abbreviations

BLS Baseline Survey

DESA SIAGA Village Alert System

DHO District Health Office

DS Desa Siaga: Village Alert System

FUS Follow up Survey

GIZ German International Cooperation

GTZ German Technical Cooperation

HMIS Health Management Information System

HR Human Resources

HRD Human Resources Development

HRIS Human Resources Information System

IHPB Integrated Health Planning and Budgeting

MNCH Maternal, Neonatal and Child Health

MoM Minutes of Meeting

Monev Monitoring and Evaluation

NTB Nusa Tenggara West

PAF Policy Analysis and Formulation

P2KT Perencanaan Penganggaran Kesehatan Terpadu: Integrated Health Planning & budgeting

P/DHA Provincial/District Health Account

PHO Provincial Health Office

PKM Puskesmas or Health Centre

PUSKESMAS Health Center

QI Quality Improvement

QI-A Quality Improvement Action

RBM Result Based Monitoring

RKA Rencana Kerja Anggaran: Budget Work Plan

SISKES Sistem Kesehatan: Health system

TOR Terms of Reference

WISN Workload Indicators Staffing Need

46

Sustainable use of products & tools in NTB province Survey ...Sustainable use of products & tools in...

Documents

Transcript of Sustainable use of products & tools in NTB province Survey ...Sustainable use of products & tools in...