Techno-Economic Assessment of

Electronic Vaccine Intelligence

Network (eVIN)

Tab

le o

f C

on

ten

tIntroduction

Objectives

Results

Recommendations

Immunization Supply ChainUniversal Immunization Program of India, largest in the world, caters its beneficiaries through ~27,000 CCPs;with 95% CCPs situated below the district level (PHC, CHC, Urban Health Facility, Sub-centers)

An Effective Immunization Supply Chain Plays a Key Role in Improving the Immunization Coverage

eVIN has been designed to provide visibility of real time stock & temperature patterns, to calculate vaccinerequirement, to tackle emergency situations of temperature breach, to provide information on consumptionpatterns and possibilities of stock reallocation

Source: VCCH 2016 Module

Rationale

• With the Gavi HSS1 support (2015-17), eVIN has been implemented in 12 out of 36 states and union territories. Gavi HSS2 support (2017-21) enabled the introduction of eVIN in the remaining 24 states and union territories. From the year 2018 onwards, state governments were expected to sustain its own implementation costs.

• Given the context, ‘Techno-economic assessment of eVIN’ was proposed to provide learnings for scale up of the progarmme in remaining states, and also pave a sustainable way forward for eventual transition of eVIN from UNDP/GAVI to GoI.

Study Objectives

01

02

03

04

To assess the programmatic usefulness of eVIN implementation, in areas of stock management and documentation, temperature monitoring and cold chain equipment

To document the program benefits and challenges of eVIN implementation, in contributing to system effectiveness and efficiencies

To evaluate the economic impact of eVIN implementation, including cost savings on vaccine and cold chain logistics management

To conduct an economic-feasibility assessment modelling the Return On Investment (ROI) of eVIN implementation

Study Design (Quasi Experimental Design)

Pre- and post- eVIN implementation assessment: To assess programmatic usefulness of eVIN

• A comparative analysis of key process indicators was done for all 12 eVIN states, based on a set of six-monthdata before eVIN implementation and another six-month data (Oct’17 to Mar’18) after eVIN implementation.

A comparative assessment in three eVIN and three Non-eVIN states:

• A comparative analysis of key performance indicators was done to assess enhanced outcomes in eVIN statesas compared with non-eVIN states.

Economic assessment of eVIN implementation :

• Economic assessment was conducted in 7 states using a pre-post study design separately for one year beforeeVIN and one year from eVIN implementation.

Both quantitative and qualitative methods were applied for programmatic assessment, while economic assessment was purely quantitative.

Total 944 cold chain points were covered in programmatic assessment. Economic assessment collected data from 102 CCPs in 7 states.

Study Design (Quasi Experimental Design)Assam

No. of Districts3 Nagaon, Dhubri,

Jorhat

No.of CCPs 44

Gujarat

No. of Districts

3 PanchmahalPorbandarKheda

No.of CCPs 124

Nagaland

No. of Districts

1 Kohima

No.of CCPs 6

Odisha

No. of Districts

3 Jajpur, Balasore, Nupada

No.of CCPs 70

Bihar

No. of Districts

4Jamui, Lakhisarai, Nalanda, Patna

No.of CCPs 63

Himachal Pradesh

No. of Districts

1 Shimla

No.of CCPs 23

Rajasthan

No. of Districts

3 BharatpurChittorgarhBikaner

No.of CCPs 132

Uttar Pradesh

No. of Districts

8

Gorakhpur, Basti, Shravasti, Varanasi, Unnao, Shahjahanpur, Kushinagar, Ghaziabad

No.of CCPs

66

eVIN States

Non eVIN States

West Bengal

No. of Districts

2 24ParganaWest Bardhaman

No.of CCPs 93

Haryana

No. of Districts

2 HisarYamunanagar

No.of CCPs 60

Manipur

No. of Districts

1 Imphal (W)

No.of CCPs 6

Jharkhand

No. of Districts

2 DumkaDhanbad

No.of CCPs 13

Chhattisgarh

No. of Districts

3 Balod, Dhamtari, Kawardha

No.of CCPs 52

Punjab

No. of Districts

3 Sangrur, JalandharLudhiana

No.of CCPs 77

Comparative States

23

15

29

10

23

ANM Health Supervisor Male

Health Supervisor Female

PharmacistOthers

eVIN States

Designation and Training Status of Cold Chain HandlersVCCH’16 Module Training

Status

10%

90%

% Trained % Untrained

eVIN States – Training Status

❖ 10 % of CCHs across 15 states belong to “others "category i.e. Class IV, Administrative, etc.

❖ Majority of CCHs are ANM and Pharmacist

ANM Health Supervisor Male

Health Supervisor Female

PharmacistOthers

Non eVIN States64

.9

1.4 9

.0 13

.1

11

.7

25%

75%

6%

94%

% Trained % Untrained

in %

in %

VCCH’16 Module Training Status

Savings in Vaccine Utilization by eVIN States (Source: Immunization Division, MoHFW)

Overall

3053

2149

Pre

Post

Uti

lizat

ion

of

Do

ses

(in

lakh

)

30% SAVING

IN VACCINE UTILIZATION

~900 lakh vaccine doses

saved

Uttar Pradesh and Bihar reported maximum saving of vaccine doses (~80 %)

• Vaccine Utilization = Opening Dosage + Dosage Received – Closing Balance • Pre eVIN period: 2015-16 for UP, MP and Raj as eVIN was implemented in early 2016-17; 2016-17- for remaining 9 states • Post eVIN period: 2016-17 for UP, MP and Raj ; 2017-18 for remaining 9 states • *Vaccines Considered -Hep-B, DPT, BCG, Pentavalent, TT : for 12 states • Measles in 11 states except HP because of introduction of MR • OPV in 9 states except Uttar Pradesh, Madhya Pradesh and Rajasthan because of t-opv to b-opv switch and mop-up rounds in 2016-17.

Facilities Reported Stock-out of any Antigen (%)

Pre Post

37.8%

26.3%

30% reduction in facilities observed stock-out

(statistically significant at 95% CI)

Facilities reported Stock-Out

Pre Post

1 instance

0.6 instance

40% reduction in instances of stock out

Instances per Facility

Pre Post

13.1 Days

8.3 Days

37% reduction in mean duration of Stock-out

Mean Duration of Stock-out

Per Facility

*Reference period: Pre: 6 months prior to inception of eVIN , post: Oct’17 to Mar’18, Vaccines-Hep-B, DPT, BCG, Pentavalent, Measles, OPV & TT for 12 states except Measles not included for HP because of introduction of MR

There is no statistically significant difference in maximum and minimum stock of any antigen after introduction of eVIN.

Percent Reduction in Missed Opportunity

Significant reduction in instances of

missed opportunities

for most of the antigens

*Missed opportunity has been calculated by number of sessions missed, multiplied by number of children immunized/facility in the pre/post period from the HMIS data for the sampled districts.

# Hep-B is in number of days of stock-out. To calculate missed opportunity, number of days of stock-out was multiplied with the number of children immunized with Hep-B in a facility in a day in the

pre/post period from HMIS data for the sampled districts. $ includes children and women

6.4

0.2

41.5

20

70.3

41.9

30.5

38.1

0

10

20

30

40

50

60

70

80

BCG HEP-B# OPV DPT Penta Measles TT$ Total

After eVIN implementation, an additional 4,01,438 antigens were administered to the due beneficiaries

in %

Facilities having Discard of any Vaccine (Source: UNDP-India reported eVIN and VCCM records)

In p

erc

en

tage

Difference in discard of any vaccine is statistically significant

30.0

22.6

Pre Post

25% reduction in facilities reporting wastage

Significance * p

Doses Discarded: Projections

Antigen Pre-eVIN Post-eVIN % reduction in doses discarded

BCG 128,751 25,750 80.0

DPT 801,670 10,668 98.7

HEP-B 64,209 33,442 47.9

MEASLES 34,278 6,688 80.5

OPV 192,626 150,489 21.9

PENTA 29,094 16,387 43.7

TT 37,121 6,688 82.0

Total 12,87,749 2,50,112 80.6

Vaccine Distribution PracticesPost-eVINPre-eVIN Post-eVINPre-eVIN Post-eVINPre-eVIN

Indent response Time# Order Fill Rate* Days left in expiry

57 %reductionIn post eVIN mean

response time was 3 days in comparison with 7 days

in pre-eVIN

No significant change was observed. In post eVIN mean

response time was 99% compared with 97% in pre-eVIN

In post eVIN mean response time was 384

days in comparison with 428 days in pre-eVIN

2 %marginal increase 10 %reduction

7 Days

3 Days

99%97%428 Days

384 Days

Record Keeping Practices

Vaccine Documentation Practices

Vaccine Stock Updating Duration

Post-eVINPre-eVIN

56.2%

97.4%

56.2%Maintaining GOI register for record

keeping has increased from

facilities in pre-eVIN to 97.4%facilities in the post-eVIN period

38.6%

53.5%

Post-eVINPre-eVIN

38.6%The daily updating of vaccine stock

register daily has improved from

facilities in pre-eVIN to 53.5%facilities in the post-eVIN period

Documentation: More Than 90% Completeness

Temperature Logbook

29%In pre-eVIN

CCPs in comparison with 70%in post-eVIN

Indent register

26%In pre-eVIN

CCPs in comparison with 69%in post-eVIN

Above 90% completeness’ indicates less than or equal to 10 instances missed out of 6 critical indicators- batch no, expiry date, VVM status, opening balance, closing balance, open vials (at CCP).

Stock Register

29%In pre-eVIN

CCPs in comparison with 75%in post-eVIN

Accuracy in Documentation (across eVIN states)

94 %accuracy matched in stock registers and eVIN entries

92 %accuracy matched in physical counting of doses and eVIN

entries

Observed Stem Thermometer Reading in the Recommended Temperature

Range at the Time of Visit

Temperature Monitoring

97.2%

81.8%

eVIN Non eVIN

CCH Recording on the Same Day

97.8% 96.5%

eVIN Non eVIN

Between 2 to 8 Degree

Accuracy of Temperature (%): eVIN Logger Matched with Stem Thermometer

88.1

6.55.4

% Match

88% accuracy

Difference is less than ± 0.5 degree Celsius

Difference between ± 0.5 to ± 1 degree Celsius

Difference is more than ± 1 degree Celsius

Cold Chain Equipment: Sickness Rate

eVIN Non eVIN

1.9%

3.4%Sickness rate among eVIN states are less

than 2%

Time Required to: Prepare Vaccine Indent, After Vaccine Receipt, and in Updating Vaccine Distribution

65 70

24

0

20

40

60

80

100

120

140

eVIN States Non-eVIN States

In M

inu

tes

Vaccine Indent

Time taken to fill register

Time taken to fill eVIN

82 82

20

0

20

40

60

80

100

120

140

eVIN States Non-eVIN States

In M

inu

tes

After Vaccine Receipt

90 90

44

0

20

40

60

80

100

120

140

eVIN States Non-eVIN States

In M

inu

tes

Updating of Vaccine Distribution

Challenges Addressed or Not

Cold Chain System MonitoringStock Management

Stock Out

Minimum Stock

Duration of Stock Out

Vaccine Indent Response Time

FEFO

Lateral Sharing

Wastage

Stock Updation

Addressed

Maximum Stock

Order Fill Rate

Not Addressed

Lack of Indents

Temperature Management

Knowledge of Temperature Trends

Temperature Record Documentation

Documentation of Breach Events

Addressed

Temperature of Cold Box

Placement of Sensor Position in Equipment

Not Addressed

Defined Key Areas of Monitoring

Addressed

Strengthening of Monitoring at Regional, District and Sub District Level

Frequent Transfer/ Change of Staff

Not Addressed

Deployment of Additional Resource (VCCM) for Monitoring

Involvement of State Officials in Monitoring

Return on Investment (INR)

Current Saving

Investment182.09 Crore

Total Saving255.84 Crore

Return On Investment1.41

Future Saving

Investment87.81 Crore

Total Saving255.84 Crore

2.91Return On Investment

Thank You!