Assessment of Malaria Outcome Indicators Using Lot Quality Assurance Sampling (LQAS):

Estimation of Bed Net Coverage in Mozambique

Caitlin BiedronASPH M&E Fellow

CDC-RwandaJanuary 15, 2009

Lot Quality Assurance Sampling (LQAS)

• Classification method used in industrial sampling to identify batches of products (or lots) with an unacceptable number of defective items.

• Typically implemented as part of a stratified random sampling design: small samples are selected from all lots in a given area.

• In each lot, the sample determines whether coverage by a health intervention exceeds a specific target by using a statistically determined decision rule.

LQAS Terminology

• A lot typically consists of a supervision area, such as a health district.

• Lots classified as being acceptable/unacceptable vis-a-vis the target.

• In this analysis lots are enumeration areas (what-if analysis).

• The decision rule is the minimal number of individuals in the sample that should have the intervention.

• Selected to determine whether a population coverage target for an intervention has been reached, such as 80% vaccination coverage.

• If this coverage target is deemed to have been reached, the lot is classified as acceptable.

LQAS Methodology

The decision rule used when applying the LQAS method can be

represented with the composite hypothesis test:

The null hypothesis will be rejected if it is determined that the coverage proportion (p) of the lot is below an adequate level of coverage (pc).

The decision rule is defined by pc , the threshold value of coverage below which program managers deem services to be unacceptable.

:

:

O c

A c

H p p

H p p

LQAS Methodology

The α error is also commonly referred to as type I error, termed the provider risk in the LQAS context. A type I error is made if one rejects the null hypothesis when it is true. It is labeled as provider risk because the provider is at risk when the health area is identified as inadequately covered by an intervention when it has in fact met the target coverage level.

The classification of substandard lots as acceptable is called consumer risk, commonly referred to as the β error or type II error. A type II error is made when one fails to reject the null hypothesis when it is false.

The magnitude of the classification error varies both with the sample size and the maximum number of units without the intervention permitted in the sample (d).

Mozambique MIS

• Mozambique MIS was conducted by the NMCP in partnership with the Malaria Consortium and PMI.

• Included a sample drawn from a subset of EAs from a population proportionate sample from the 1997 census.

• A total of 346 EAs with 5990 households were selected from the 1510 EAs of the primary sample.

• After data cleaning, a total of 5745 household records were available for analysis.

• The survey was conducted June - July 2007.

LQAS Parameters

• We based our decision rule on a coverage target (pc) equal to 70% and defined our α error to be 10%.

• Year 1 Mozambique PMI target for bed net possession called for

70% of households to own at least one ITN.

• In urban clusters, the typical sample size was 20 households. In rural clusters, the typical sample size was 15 households.

• Based on these parameters, we calculated the decision rule for both urban and rural enumeration areas.

• The decision rule for urban clusters was 12 households; for rural clusters it was 8 households.

LQAS Results: Any Bed-net (Urban EAs)

TABLE 1. LQAS results for household possession of any bed-net in Manica, Mozambique, 2007

Decision Rules set for α<0.10

Cluster ID No Yes Total Decision Rule LQAS

175 11 8 19 11 failure

176 6 14 20 12 success

177 12 8 20 12 failure

178 8 12 20 12 success

179 8 11 19 11 success

180 10 10 20 12 failure

181 2 15 17 10 success

182 6 14 20 12 success

183 15 5 20 12 failure

184 8 12 20 12 success

185 7 14 21 12 success

186 11 9 20 12 failure

187 7 13 20 12 success

LQAS Results: Any Bed-net

TABLE 3. Coverage proportions, confidence intervals and LQAS result summaries for household possession of any bed-net in Mozambique

Province Coverage Proportion 95% Confidence Interval EAs w/adequate coverage

 

Niassa 0.406 (0.323, 0.489) 12 (34)

Cabo Delgado 0.376 (0.305, 0.448) 8 (34)

Nampula 0.300 (0.247, 0.353) 5 (36)

Zambezia 0.310 (0.243, 0.377) 6 (36)

Tete 0.318 (0.241, 0.394) 9 (34)

Manica 0.427 (0.363, 0.492) 12 (28)

Sofala 0.512 (0.445, 0.579) 18 (34)

Inhambane 0.315 (0.253, 0.377) 10 (34)

Gaza 0.368 (0.299, 0.437) 6 (24)

Maputo Province 0.335 (0.281, 0.389) 3 (32)

Maputo Cidade 0.481 (0.416, 0.546) 5 (20)

LQAS Results: Any ITN

TABLE 4. Coverage proportions, confidence intervals and LQAS result summaries for household possession of any ITN in Mozambique: 2007

Province Coverage Proportion 95% Confidence Interval EAs w/adequate coverage

Niassa 0.178 (0.130, 0.226) 0 (34)

Cabo Delgado 0.195 (0.134, 0.256) 3 (34)

Nampula 0.084 (0.053, 0.116) 0 (36)

Zambezia 0.143 (0.084, 0.201) 2 (36)

Tete 0.130 (0.081, 0.180) 1 (34)

Manica 0.368 (0.301, 0.435) 5 (28)

Sofala 0.228 (0.171, 0.284) 2 (34)

Inhambane 0.102 (0.072, 0.133) 0 (34)

Gaza 0.119 (0.072, 0.165) 0 (24)

Maputo Province 0.067 (0.042, 0.092) 0 (32)

Maputo Cidade 0.099 (0.068, 0.131) 0 (20)

Mozambique Bed-net Coverage

LC-LQAS Background

• May not be feasible to conduct LQAS in the traditional fashion; instead only a subset of the areas in the catchment region represented would be selected.

• Pooled data would no longer be a stratified random sample, but instead a cluster sample, if the areas had been chosen at random.

• The recently developed large-country lot quality assurance sampling (LC-LQAS) method is used to integrate LQAS with cluster sampling.

• We next investigated the possibility of taking only a sub-sample of the EAs included in each province to calculate the provincial and national coverage estimates.

• This second aggregation was done in order to compare the estimates obtained using a smaller sample (220 EAs) to those estimates resulting from the full sample (346 EAs).

LC-LQAS Results: Any Bed-net

TABLE 5. LC-LQAS sub-sample coverage proportions, confidence intervals and LQAS result summaries for possession of any bed-net

Province Coverage Proportion 95% Confidence Interval EAs w/adequate coverage

Niassa 0.395 (0.272, 0.518) 5 (20)

Cabo Delgado 0.395 (0.289, 0.500) 4 (20)

Nampula 0.316 (0.243, 0.390) 1 (20)

Zambezia 0.318 (0.210, 0.426) 3 (20)

Tete 0.355 (0.224, 0.486) 6 (20)

Manica 0.447 (0.373, 0.520) 9 (20)

Sofala 0.479 (0.390, 0.568) 10 (20)

Inhambane 0.350 (0.241, 0.459) 7 (20)

Gaza 0.374 (0.295, 0.452) 6 (20)

Maputo Province 0.330 (0.265, 0.394) 2 (20)

Maputo Cidade 0.481 (0.416, 0.546) 5 (20)

LC-LQAS Results: Any ITN

TABLE 6. LC-LQAS sub-sample coverage proportions, confidence intervals and LQAS result summaries for household possession of any ITN in Mozambique: 2007

Province Coverage Proportion 95% Confidence Interval EAs w/adequate coverage

Niassa 0.128 (0.077, 0.180) 0 (20)

Cabo Delgado 0.222 (0.123, 0.321) 1 (20)

Nampula 0.067 (0.026, 0.107) 0 (20)

Zambezia 0.158 (0.051, 0.266) 1 (20)

Tete 0.149 (0.041, 0.257) 0 (20)

Manica 0.402 (0.324, 0.481) 4 (20)

Sofala 0.161 (0.108, 0.214) 2 (20)

Inhambane 0.103 (0.059, 0.146) 0 (20)

Gaza 0.135 (0.085, 0.186) 0 (20)

Maputo Province 0.067 (0.034, 0.100) 0 (20)

Maputo Cidade 0.099 (0.068, 0.131) 0 (20)

Comparison of Coverage Estimates

TABLE 7. Provincial and national coverage estimates for aggregate LC-LQAS, LQAS, and MIS cluster-samples: Mozambique 2007

Province Any Bed-Net in HH Any ITN in HH

 LC-LQAS sub

samplefull LQAS

sampleMIS Cluster

estimateLC-LQAS sub

samplefull LQAS

sampleMIS Cluster

estimate

Niassa 0.395 0.406 0.422 0.128 0.178 0.177

Cabo Delgado 0.395 0.376 0.378 0.222 0.195 0.196

Nampula 0.316 0.300 0.329 0.067 0.084 0.087

Zambezia 0.318 0.310 0.365 0.158 0.140 0.178

Tete 0.355 0.318 0.317 0.149 0.130 0.119

Manica 0.447 0.427 0.448 0.402 0.368 0.369

Sofala 0.479 0.512 0.504 0.161 0.228 0.217

Inhambane 0.350 0.315 0.323 0.103 0.102 0.112

Gaza 0.374 0.368 0.373 0.135 0.119 0.133

Maputo Prov. 0.330 0.335 0.297 0.068 0.067 0.057

Maputo Cidade 0.481 0.481 0.486 0.099 0.099 0.102

National Coverage 0.365 0.355 0.375 0.143 0.145 0.158

Strengths of LQAS

• Allows for pass/fail classification based on various decision rules; • Dichotomous nature of the results can provide local supervisors with

a decisive judgment about action to be taken; • Sample sizes are typically smaller than those required to perform

other estimation analyses;• Inexpensive relative to traditional surveys;• Surveys can capture variability across local areas, which

encourages local level monitoring and accountability;• Surveys can be conducted more often, providing information more

frequently than HH surveys;• Sample can be treated as a stratified sample for further analysis;• With adequate sampling and appropriate weighting, can provide

estimates at aggregate levels (provincial, national);• To accommodate large countries, sampling areas may be chosen

randomly based on cluster sampling methodology (LC-LQAS).

Limitations of LQAS

• A high level of technical competence is required to achieve more than the basic pass/fail classification;

• Understanding of sampling frame requires knowledge of household selection, weighting and binomial probability;

• Perception that methodology is ‘simple’ belies technical complexity;• Decision rules may seem arbitrary if they are not based on explicit

program targets or certain thresholds related to transmission;• Sample sizes tend to be small, so estimates may have large

confidence intervals when sample combined;• Pass/fail outcomes do not give information on scale of change at the

local level, and it is not clear how dichotomous outcomes will be used to track progress in programs;

• Survey EAs must be selected in accordance with health districts or administrative units in order for results to have programmatic relevance.

Potential Applications in the Field

• The demonstrated methods could play a tangible role in the field where there is a growing interest in obtaining coverage estimates for small geographic units.

• LQAS could be used to assess EA data collected by standard MIS surveys to provide more frequent tracking of coverage indicators for districts country-wide or within key target areas.

• The standard MIS methodology requires large sample sizes to obtain reliable district level point estimates, making the survey both cumbersome and costly to conduct.

• LC-LQAS could provide local and national level estimates for coverage when it replaces the standard MIS sampling.

• LC-LQAS methodology would be applicable in those countries in which survey sampling units are selected in accordance with health districts or administrative units.

Potential Applications in the Field

• LQAS offers an alternative to malaria control program managers who are interested in tracking coverage at a local level to improve their service delivery strategies and to adjust priorities.

• As more countries aim to control malaria and scale-up

coverage to do so, it may become necessary to offer alternatives to the national level surveys that provide a single set of indicators for a country and do not present differences across local levels.

• National malaria programs are increasingly interested in

getting data from smaller geographic areas; LQAS may be a suitable method for such programs to use.

Acknowledgements

Amy Ratcliffe, CDC Malaria BranchAlbert Kilian, Malaria ConsortiumMarcello Pagano, Harvard School of Public HealthMarcia Castro, Harvard School of Public HealthBethany Hedt, Harvard School of Public HealthJoe Valadez, Liverpool School of Tropical MedicineJuliette Morgan, CDC-Mozambique