Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
-
Upload
ravi-shankar -
Category
Documents
-
view
215 -
download
0
Transcript of Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
1/27
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
2/27
ACCURACY ASSESSMENT OF POST - EARTHQUAKE
BUILDING DAMAGE CLASSIFICATION IN HAITI
Master of Disaster Management (MDMa)Masters Thesis 200910
RAVI SHANKAR - Architect - Planner
Supervisor: Asst. Prof. Peter Kjr JensenCo-Supervisor: Michael Calopietro
Institute of International Health, Immunology & Microbiology
Faculty of Health Sciences, University of Copenhagen
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
3/27
My parents, my wife, and my son for their love and support &
to the Haitians who are waiting patiently.
I am grateful to my supervisor, Asst. Prof. Peter Kjr Jensen, a man who waves a magic wand. It
was a great pleasure working with my Co- Supervisor, Michael Calopietro, who made it possible
for me to write my thesis in absentia, away from Copenhagen.
It is an honour to work with the United Nations Operational Satellite Applications Programme
(UNOSAT) for my thesis and would like to thank our director, Francesco Pisano, and my
managers, Dr. Einar Bjorgo and Oliver Senegas, for their extended guidance. My friend Michael
Jendryke needs a special appreciation for his constant support and valuable input. I am indebted
to my colleagues Luca DellOro, Joshua Lyons, Wendi Pedersen, Fana Woldegebreal, Karl Nei-
man, and Paola, Fred and Oliver Vandamme for their constant help as well. This thesis would not
have been possible without assistance from Centre National de LInformation Go-Spatiale- Haiti
(CNIGS) in eld data collection, which was one of the most difcult parts of the whole exercise.
I would also like to thank Dr. Norman Kerle (ITC, Netherlands) and Dr. Keiko Saito (Cambridge
Architectural Resea rch Limited) for their valuable insi ghts. I would like to show my gratitude to
European Commission-Joint Research Centre (EC-JRC) and World Bank (WB) for sharing their
data. Finally, it is a pleasure to thank my University Department staff, classmates, and friends
from Danida Fellowship Centre, and all others who made this thesis possible.
Acknowledgements
to
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
4/27
Quantitative measurements of building damage are extremely important after an earthquake. An
integrated remote sensing and GIS-based rapid mapping approach can provide timely informa-
tion on building damages to be used by the international humanitarian community in planning aid
response. The unprecedented scale of damages wreaked by the 2010 earthquake in Haiti, com-
pounded by the difcult social and political conditions there, precluded widespread eld damage
assessment. Therefore, most of the preliminary damage assessments were carried out throughremote sensing images. Remote sensing imagery techniques are indispensable where difcult
ground conditions prevail, but there are limitations as well. More than 2000 maps were published
by different agencies within 75 days after the Haiti disaster. During this period, the variance in
the spatial data produced by these different agencies could have compromised relief efforts.
Naturally, then, there was a sense of urgency in assessing the accuracy of the information and
the methods used to obtain it.
This thesis investigates the accuracy of photo interpretation, one of the methods used to conduct
the virtual building damage assessment by the three main agencies involved: UNOSAT, World
Bank, and European Commission-Joint Research Centre. An error(confusion) matrix was used
to evaluate the accuracy of these assessments. The analysis was carried out to check overall ac-
curacy , accuracy per communes, accuracy per damage classes, and accuracy per each agency.
Finally, the assessment errors were investigated in detail in order to identify their causes, which
helped to quantify the limitations of using the remote sensing images. The results show thatthe overall accuracy was around 60% and the range varied between 36-82% among different
communes. Overall accuracy was higher with fewer numbers of damage classes. Post-mortem
analysis of errors suggests that there is a strong possibility the accuracy levels can be improved.
Abstract
5
source: google earth
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
5/27
Aerial photography Aerial photography is the taking of photographs of the ground from an elevated position. The term
usually refers to images in which the camera is not supported by a ground-based structure.
CNIGS Centre Nationale de lInformation Geo Spatiale
EC-JRC European Commission - Joint Research Centre
Damage assessment An assessment of the total or partial destruction of physical assets, both physical units and
replacement cost
ECLAC Economic Commission for Latin America and the Caribbean
GEER Geo-Engineering Extreme Events Reconnaissance
GIS Geographic Information System (GIS), or geographical information system, is any system
that captures, stores, analyzes, manages, and presents data that are linked to location. In
the simplest terms, GIS is the merging of cartography and database technology.
Loss assessment An analysis of the changes in economic ows that occur after a disaster and over time, valued at
current prices.
Housing damage assessment A damage assessment that analyzes the impact of the disaster on residential communities, living
quarters, and land used or housing
Rapid assessment An assessment conducted soon after a major event, usually within the rst two weeks. It may
be preceded by an initial assessment, and it may be multi-sector or sector-specic. It provides
immediate information on needs, possible intervention types, and resource requirements.
RS Remote sensing (RS) is the small- or large-scale acquisition of information of an object or
phenomenon through the use of either recording or real-time sensing device(s) that are
wireless, or not in physical or intimate contact with the object (such as by way of aircraft, spacecraft,
satellite, buoy, or ship).
UNOSAT UNOSAT is the UN Institute for Training and Research (UNITAR) Operational Satellite
Applications Programme, implemented in cooperation with the European Organization for
Nuclear Research(CERN).
WB World Bank
Abbreviations and keywords
7
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
6/27
1 INTRODUCTION
1.1 Background ..................... ...................... ..................... ...................... ...................... ...... 11
1.2 The Earthquake ..................... ...................... ..................... ...................... ...................... 12
1.3 The Critical Period................. ...................... ...................... ...................... ...................... 13
1.4 Post-Disaster Needs Assessment............. ...................... ...................... ...................... .. 14
1.5 Role of GIS in Disaster Management .................... ...................... ...................... ........... 15
1.6 Disaster Management and Remote Sensing ..................... ...................... ..................... 15
1.7 Use of RS Images after Haiti Earthquake .................... ...................... ...................... ..... 16
1.8 Methods of Damage Assessments........... ...................... ...................... ...................... ... 17
1.9 Damage Catalogue................ ...................... ...................... ...................... ...................... 18
1.10 Statement of Problem.................. ...................... ...................... ...................... .............. 19
2 OBJECTIVES
2.1 Main Objectives........... ...................... ...................... ...................... ...................... .......... 21
2.2 Hypothesis............. ...................... ...................... ..................... ...................... ................. 21
2.3 Specic Objectives ..................... ...................... ...................... ...................... ................ 21
3 METHODS
3.1 Introduction .................... ...................... ...................... ...................... ...................... ....... 23
3.2 Sample Selection ...................... ...................... ...................... ...................... .................. 24
3.3 Data Collection .................... ...................... ...................... ...................... ...................... . 25
3.4 Quality Assurance.............. ...................... ...................... ...................... ...................... .... 25
3.5 Methods for Accuracy Assessments ...................... ...................... ...................... ............ 26
3.6 Creation of Database and Analysis Approach ................... ...................... ...................... 27
4 FINDINGS
4.1 Overall Findings............ ...................... ..................... ...................... ...................... .......... 31
4.2 Variations by Communes................. ...................... ...................... ...................... ............ 33
4.3 Four-Class and Two-Class Variations ...................... ...................... ...................... ......... 34
4.4 Postmortem of Failed Class ..................... ...................... ...................... ...................... ... 35
5 DISCUSSION
5.1 Methodology.................... ...................... ...................... ...................... ...................... ...... 37
5.2 Overall Communes ...................... ...................... ...................... ...................... ............... 38
6 CONCLUSION & RECOMMENDATIONS 41
7 BIBLIOGRAPHY 43
8 ANNEXURE 45
Figure 1 Map of Haiti.
Figure 2 Area of populations affected by the 2010 earthquake.
Figure 3 Relief, Response, Recovery and Reconstruction phases as described by World Bank.
Figure 4 Sequence of the Post-Disaster Needs Assessments.
Figure 5 Published maps after 2010 Haiti earthquake.
Figure 6 Various methods of damage assessments.
Figure 7 Damage catalogue as used by various agencies for damage assessments in Haiti.
Figure 8 The pre- and post-images with faulty building damage assessments.
Figure 9 Showing various damage classes (destroyed to no visible damage) as adopted by UNOSAT.
Figure 10 Percentage composition of the assessment among different agencies.
Figure 11 Methodologies of accuracy assessments.
Figure 12 Showing random samples collected for eld validation.
Figure 13 Showing the theory of error matrix
Figure 14 Showing the methodology adopted for accuracy assessment.
Figure 15 Showing comparison on four and two class for various agencies.
Figure 16 Showing kappa statistics for various communes.
Figure 17 Showing kappa statistics for various agencies
Figure 18 Showing accuracy levels comparison between four and two class.
Figure 19 Showing logic behind post mortem of failed class.
Table 1 Buildings Assessed in Various Communes.
Table 2 Field Samples Collected by CNIGS from Various Communes.
Table 3 Overall ndings .
Table 4 Range on Kappa Statistics Agreement
Table of contents List of figures
List of tables 9
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
7/27
1.1 Background
Haiti is located in the Caribbean Sea on the Island of Hispaniola, which it shares with the Domini-
can Republic (Fig. 1). Haiti occupies the western third of the island and has 8.4 million inhabitants
and a population density of 300 people per km2 (Canadian International Development Agency,
2005). Haiti is the poorest country in the Western Hemisphere with an estimated 80% of its people
living under the poverty line; 54% live in abject poverty (CIA, 2010). When it comes to natural dis-
asters, Haiti seems to have a bulls-eye on it. There were eight major natural disasters within last 15
years. Floods during 2008 hurrican e killed more than 2500 people.
Haiti is a twentieth-century Greek tragedy of Malthusian prophecy run wild, and little time remains
to reverse the course toward destruction of what once was the lush tropical beauty of the Pearl of
the Antilles (Ernest H, 1996). High population density, chaotic urbanizatio n, a weak political struc-
ture, and poorly developed disaster-management systems are some of the factors contributing
to the vulnerability of its people.
Figure 1. - Map of Haiti
11
INTRODUCTION
1
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
8/27
1.2 The Earthquake
The earthquake that struck the Republic of Haiti on January 12, 2010, is one of the most signifi-
cant earthquakes in recorded history. Destruction and death on a huge scale unfolded across
the densely populated Haitian capital, Port-au-Prince, and into the shanties and villas of the sur-
rounding mountains. As of February 16, 2010, the death toll reported by the government of Haiti
exceeded 217,000 with an additional 300,000 injuries. More than 5 million people live in the area
directly affected by the earthquake (Fig. 2), and 1.2 million people are now living in temporary
shelters (UN, 2010).
Most of the buildings are masonry either brick or construction blocks which tend to perform badly
in an earthquake and with sub standard construction techniques worsened the situation. Damage
to infrastructure was extensive and ministry of education estimated that more than 15,000 primary
schools and 1500 secondary schools were severely damaged. Most of Port-au-Princes municipal
government buildings were destroyed or heavily damaged in the earthquake, including the City
Hall, which was described by the Washington Post as, a skeletal hulk of concrete and stucco,
sagging grotesquely to the left(Manuel Roig-Franzia, 2010).
1.3 The Critical Period
After t he ea rthquake, the government of H aiti led t he emerge ncy reli ef operat ion with su pport
from the international community. Preliminary physical damage assessments on buildings were
out within days after the earthquake from various agencies. In its handbook on disaster man-
agement, recovery, and reconstruction, the World Bank describes the initial 25 days as the most
crucial period for disaster relief and damage loss assessment (Fig. 3; ECLAC & World Bank, 2003).
Major response guidelines from the World Bank suggest that the following activities take place
during the initial phase,
Disaster Response
Disaster Relief
Disaster and loss assessments
The decisions on reconstruction in Haiti were based on these timely assessments along with the
updated field results.
Figure 2.- Area of populations affected by the 2010 earthquake.
Figure 3. Relief, Response, Recovery and Reconstruction phases as described by World Bank.
13
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
9/27
1.4 Post-Disaster Needs Assessment
The Post-Disaster Needs Assessment and Recovery Framework (PDNA) is an interagency mecha-
nism set up to facilitate a comprehensive, coordinated, and efficient international approach to
damage, loss, and needs assessment and recovery planning. The Recovery Framework outlined
by the PDNA facilitates integrity of damages data and helps to ensure that financing needs and
the recovery pathway are matters of mutual agreement. PDNA is designed to address one or
more of the following issues (Darcy and Hofmann, 2003):
Whether to intervene.
The nature and scale of the intervention.
Prioritization and allocation of resources.
Programme design and planning.
PDNA is not intended to replace more in-depth assessments, which may still be required outside
of the PDNA for refining the understanding of needs, generating response options, and develop-
ing recovery projects within a given sector. In Haiti, the PDNA is a government-led exercise, with
integrated support from the United Nations, the European Commission, the World Bank, and other
national and international partners. A PDNA prepares a single consolidated report containing
crucial information on the disaster: its physical impacts; the economic value of the damages
and losses; the human impacts as experienced by the affected populations; and the early and
long-term recovery needs and priorities. The type and scope of the disaster affects the develop-
ment vision. Based on this assessment, various responses are proposed, analyzed, and selected
for inclusion in the Recovery Framework (Fig. 4).
1.5 Role of GIS in Disaster Management
Geographic Information System (GIS) is a computerized method of handling the spatial data,
which includes in formation system and remot e sensing data. In simple terms, GIS is a mergi ng of
the cartography and database technology. GIS is significant to all stages of the disaster manage-
ment cycle. It provides for efficient and rapid management of data, as well as data manipulation
and analysis, thereby effecting better decision making. GIS can be an essential support system
to estimate the loss after a disaster. In the present context of Haiti, humanitarian organizations
fully exploited the role of GIS in disaster management. Haiti represents a milestone in the use
of GIS in the post-disaster cycle, as in agencies relied upon it more than ever before to collect,
process,and analyze the spatial information gathered.
1.6 Disaster Management and Remote Sensing
Remote sensing (RS) is the small- or large-scale acquisition of information of an object or phe-
nomenon using either recording or real-time sensing devices that are wireless, or otherwise not in
physical or intimate contact with the object (such as by way of aircraft, spacecraft, satellite, buoy,
or ship) (Lillesand et al., 2003). The critical period after the disaster is the response phase, and
the field data collected during this period, if it is possible to do so, is very valuable. However, it is
a difficult task to collect data during this period from the ground. But remote sensing technology
can accommodate this lag in collecting field data.
Using remote sensing technology, the damage concentration in the urban areas can be located
in a shorter time compared with the conventional ground survey (Rao, 2000). An unprecedented
international satellite imagery campaign with the participation of all leading Earth observation
operators was undertaken after the destructive Haiti earthquake. Annexure 1 shows the various
remote sensing accusations that took place after the earthquake. Later, high-resolution aerial im-
ages of the capital city of Port-au-Prince of up to 15 cm spatial resolution were taken for more pre-
cise assessments of building stability. Never before have such high- resolution spatial images and
aerial photography been so accessible. Multi-source data were collected from various missions:
WB-Image CAT-RIT Remote Sensing Mission (LiDAR optical data, 15 cm & 2 pt/m), Google (optical
data, 15 cm), NOAA (optical data, 25 cm), and Pictometry (Haiti_Reconnaissance.pdf, 2010).
Figure 4. Sequence of the Post-Disaster Needs Assessments.
15
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
10/27
1.7 Use of RS Images after Haiti Earthquake
There were 2000+ maps* published within the first 75 days of the Haiti earthquake. More than
50 agencies were directly involved in the preparation and publication of the maps. Most of the
maps detailed earthquake coverage, affected populations, and damage assessments, and
some offered hazard analysis and cluster- or sector related data. There were around 10 agencies
involved in the building damage assessments. With the Post-Disaster Needs Assessment requir-
ing building damage extents for Haiti, these assessments happened to be a valuable exercise.
The UN Institute for Training and Research (UNITAR), the Operational Satellite Applications Pro-
gramme (UNOSAT), the European Commission -Joint Research Centre (EC-JRC), the National
Geospatial Information Centre (CNIGS) representing the Haiti Government, the World Bank (WB),
and the Global Facility for Disaster Reduction and Recovery (GFDRR), supported by its consult-
ant Image CAT, have carried out a detailed, building-by-building assessment of the damage.
This analysis relied heavily on the use of space remote sensing techniques. The joint agreement
among UNOSAT, WB, and EC-JRC to publish a common database for all of Haiti must have been
a welcome relief for the humanitarian community.
1.8 Methods of Damage Assessments
Participating agencies used different approaches in assessing the extent of building damage.
Predominant among them was the photo interpretation method. Figure 6 shows various methods
of damage assessment made possible by remote sensing images. Photo interpretation method
is theoretically one of the simplest methods of building damage assessment using RS images.
The analyst uses GIS software where the pre- and post- satellite/aerial images are loaded, and a
new database is created to record the assessments. The analyst visually identifies the extent of
damage by comparing the pre- and post- images and records it into the database. Though the
method sounds simple, it is time consuming and labour intensive. And there are drawbacks to
the technology, as described by UNOSAT: Satellite imagery generally provides an overhead view-
point. Lateral damage and especially damage to the internal structures of buildings, is not de-
tectable from the analysis of satellite imagery (2010). In the manual photo interpretation method,
the quality of the assessment might depend on the expertise of image analysts working away from
Haiti, who are better equipped for the task if they are knowledgeable about Haitis construction
techniques, building materials, and housing typology...
Figure 5. Published maps after 2010 Haiti earthquake.
Figure 6. Intensity of Damage assessment as published by UNOSAT.
* Authors estimate based on the relief web website (www. reliefweb.int)
Figure 7. Relief, Response, Recovery and Reconstruction phases as described by World Bank.
17
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
11/27
1.9 Damage Catalogue
The important aspect of building damage assessment is the expression of the damage levels.
The damage catalogue is the scale which portraits the intensity of the damage, with categories
usually ranging from no visible damage to destroyed. There are several international field dam-
age assessment scales (e.g.European Macroseismic Scale-98[EMS-98], ACT ) that quantify the
extent of damages after an earthquake. It is very clear from the published maps of Haiti that the
participating agencies used different damage catalogues for assessment. UNOSAT and EC-JRC
used four classes in its damage assessments whereas WB used just two classes for the damage
assessment (Fig. 8).
Figure 8. Damage catalogue as used by various agencies for damage assessments in Haiti.
Figure 9. Showing various damage classes (destroyed to no visible damage) as adopted by UNOSAT.
A- UNOSAT , B- EC-JRC/ISFEREA, C- DLR,GERMAN SPACE AGENCY, D- SERTIT E- ITHACA & E-Geos
Figure 10. The pre- and post-images with faulty building damage assessments.
1.10 Statement of Problem
With fast spreading application o f GIS using remotely sensed data and ever increasing evaluat ion
of the damages using these technologies, it is essential that the accuracy of these products be
assessed. Accuracy assessments evaluate either a single method or various methods used to col-
lect spatial data in the aftermath of the disaster. The accuracy assessment allows the analyst to
quantitatively compare different methods. . The cross comparisons should identify the processes
that yield the most accurate results, information critical for decision making in future disasters.
The ultimate objective should be to make accuracy assessment part of the metadata in the later
updates of published maps. Most of the maps produced after the disaster as rapid mapping or
damage assessment had not been subject to extensive accuracy assessments. It appears there
was often a casual attitude toward these maps, in part because they had been prepared so hast-
ily in the first place. In some instances, it seemed the appearance of the map was valued over
its accuracy.
19
The sample validation in field by GEER (Geo -
engineering Extreme Events Reconnaissance)
stated However, it must also be accepted with
some limitations as these types of surveys do not
always capture the magnitude of destruction on
the ground. As an isolated example, consider
the two steel frame warehouses ........ Both of
these buildings were heavily damaged and will
need to be torn down, yet they were classified
as No Visible Damage by the UNOSAT survey.
Ellen Tathje, Jeff Bachhuber, et, all, GEOTECHNICAL
ENGINEERING RECONNAISSANCE OF THE 2010 HAITI
EARTHQUAKE, 22 Feb 2010.
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
12/27
OBJECTIVES
2
2.1 Main Objectives
The aim is to evaluate the accuracy of photo interpretation methods of remote/virtual building
damage assessment in the aftermath of the Haiti earthquake. The study will focus on the assess-
ments of the three main coordinating agencies: UNOSAT, EC-JRC, and WB.
2.2 Hypothesis
There is no difference in accuracy of preliminary damage assessments among different com-
munes and agencies for the Haiti earthquake.
2.3 Hypothesis
. To evaluate the overall accuracy of building damage assessments for Haiti.
. To compare the accuracy of building damage assessments by commune.
. To compare the accuracy levels among the various agencies.
. To compare the accuracy levels of the four and two-class building damage classification.. To perform a post-mortem analysis of the errors of the building damage assessment.
21
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
13/27
METHODS
3
3.1 Introduction
There were around 400,000 building points in Haiti which had been under the direct influence of
the Jan 12th earthquake. The study area encompasses communes of the Artibonite region. In
these, Port-au-Prince, Cite-Soleil and Carrefour are the most densely populated areas with the
maximum population spread. It is also the area of the greatest economic development in Haiti. In
total, 290,672 building points were assessed by these three organizations. UNOSAT covered 83%
of the total assessment with EC-JRC responsible for about 12 % and WB for the remaining 5 % (Fig.
9). World Bank assessed only the Destroyed and Severely damaged classes whereas UNOSAT and
EC-JRC assayed four classes, Destroyed, Severely Damaged, Moderate Damage, and No Visible
Damage.
NAME OF COMMUNE NO OF BUILDINGS ASSESSED ORGANIZATION
CROIX-DES-BOUQUES 3562 WORLD BANK
CITE-SOEIL 6709 WORLD BANK
CARREFOUR 69384 UNOSAT
LEOGANE 23790 EC-JRC
DELMAS 64394 UNOSAT
TABARRE 3919 WORLD BANK
PETIN- VILLE 10587 UNOSAT / WORLD BANK
GRESSIER 1347 WORLD BANK
PORT-AU-PRINCE 92432 UNOSAT
JACMEL 13133 EC-JRC
PETIT-GOAVE 409 WORLD BANK
GRAND-GOAVE 273 WORLD BANK
TOTAL 290672
Table 1 Building Assessed in Various Communes.
Figure 11. Percentage composition of the assessments among
the different agencies.
EC-JRC
36284
12%
WB136865%
UNOSAT
240702
83%
23
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
14/27
3.2 Sample Selection
The earthquake affected areas of Haiti was divided into 200X200 m cells. To be considered for a
sample selection, each cell had to satisfy certain criteria. Cells were sorted based on the follow-
ing three steps:
1.Each cell was assigned either high or low density as per existing population.
2. Each cell was also assigned high or low damage intensity based on damage assessment.
3. A land cover map was relied upon for uniform representation within each land use.
Analysis using these criteria identified around 2150 cells from whic h samples were t o be cho-
sen. A stratified sampling method was then used to further refine the sample-cell collection. To
determine the sample size to be selected from the 2150 retained cells, confidence level and
confidence interval were defined as prior criteria and calculations were performed accordingly.
A 95% confidence level a nd a 10.5 confidence interval were select ed. Hence , the computed
sample size was equal to 86 cells, representing 4% of the population. Finally, the 86 cells selected
randomly among the 2150 previously stratified cells in a number proportional to the stratums size.
A field quest ionnaire was prepared on the basis of European Macroseismic Scale-98, and every
building from the chosen sample was graded accordingly.
3.3 Data Collection
In total, around 7000 building points were collected in the field by CNIGS, who were entrusted
with t hat work. Samples from the co mmunes of Jacmel, Petit Goane, and Grand Goane were
not collected for various reasons, including damages, accessibility, and distance from the base
station. Even though little pre-processing was undertaken in the site office at Haiti, this was one of
the most time consuming components of the whole project.
These also include the samples collected by UNOSAT/EC-JRC during the initial eld visit to Haiti.
Table 2 Field samples Collected by CNIGS from Various communes.
25
NAME OF COMMUNE NO OF FIELD SAMPLES NO OF BUILDINGS
ASSESSED VIRTUALLY
CROIX-DES-BOUQUES 343 3562
CITE-SOEIL 243 6709
CARREFOUR 1305 69384
LEOGANE 97 23790
DELMAS 735 64394
TABARRE 648 3919
PETIN- VILLE 464 10587
GRESSIER 50 1347
PORT-AU-PRINCE 2607 92432
CNIGS FIELD DATA 6493
UNOSAT/ EC-JRC FIELD DATA 442
3.4 Quality Assurance
Field samples were scrutinised during the preprocessing phase for quality assurance. The preproc-
essing for error checking included:
1. Sample points outside the building boundary or on the road
2. More than one sample point within the same building or more than one virtual assess-
ment point for a given building
3. Site photos giving extreme class variation in comparison to field assessment.
After the in itial preprocessing of the sample for possible errors, the size of the bui lding points wasreduced to 6492. This covers roughly 2% of virtual assessed building samples.
Figure 12. Showing random samples collected for eld validation.
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
15/27
Figure 13. Showing the theory of error matrix
3.5 Methods for Accuracy Assessments
To evaluate the accuracy levels of the remote damage assessments an error( confusion) matrix
is used. It compares category-by-category, based on damage class/catalogue, the relationship
between the produced and known reference data of field truth. Discrete Multivariate analysis
technique is used following the error matrix with the assumption that these categories are de-
pendent and do not require any transformation. A Khat value (K^) will be calculated for each
agency, which is a measure of how well the classification agrees with the field data. Khat can
be tested using the normal curve deviate to determine if the two error matrixes are significantly
different.
3.6 Creation of Database and Analysis Approach
The following steps were carried out for data analysis.
Step 1: The joint database was created by UNOSAT combing the individual data sets of
UNOSAT, EC-JRC, and WB.
Step 2: The data received from the field was subjected to preprocessing as explained
before. After removing the field samples which did not meet the set criteria,
a new field database was created.
Step 3: The joint database was then merged with the field database. This created an
intermediate data base.
Step 4: The intermediate database was further divided into shape files as per communes
for 4-class and 2-vclass building classification.
Step 5: Accuracy assessment script was run using ESRI ARCGIS 9.3.1 software followed
by Kappa statistics script.
Step 6: The results are displayed and postmortem of the failed class is then carried out.
27
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
16/27
Figure 14. Showing the methodology adopted for accuracy assessment.
29
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
17/27
FINDINGS
4
4.1 Overall Findings
Out of the 5806 points organized after preprocessing the field data, there were only 4351 virtu-
ally assessed points matching with the field points. This means that more than 1455 points were
not assessed during the virtual building assessment. Thus, for Haiti, the virtual/remote building
damage assessment was performed for only 75% of the total buildings that were affected within
the earthquake impact zone. A number of factors may have contributed to this oversight. Therewere si gnificant problems defining the various classes of buildings and indeed what constitut-
ed a building, especially given the makeshift character of many of the settlements. A number
of conditionals were exacted upon the labelling causing further confusion. Also, there was a
troubling lack of pre-earthquake data establishing building parcel boundaries. One could ar-
gue that this is generally the case with most of the disasters that occur in developing or under-
developed countries. Had building parcel boundaries been available, damages could have
been associated with the building parcel rather than with the building itself. Some of the screen
shots illustrate that individual analysts had clearly failed to mark the buildings in most instances.
31
CROIX-DESBOUQUES 35 11.5 31 63 10.5 25 29 11
CITE-SOEIL 44 25 38 72 20.29 21 20 15
CARREFOUR 67 37 38 79 21.28 12 9 12
LEOGANE 46 40 38 78 25.40 25 18 9
DELMAS 82 31 40 94 20.93 6 8 4
TABARRE 76 32 36 92 26.46 11 6 7
PETINVILLE 61 24 25 94 - 14 12 13
GRESSIER 56 - 35 76 12.73 25 13 8
PORT-AUPRINCE 56 39 40 84 25.29 20 14 9
OVERALL COMMUNES 61 36 39 86 23.06 - - -
NAME OF COMMUNE
4 class Post mortem of errors
Accuracylevels
Usersaccuracy
%
Producersaccuracy
%
2 classAccuracy
levels %
Kappastatistics
%
Errorsby
analyst %
Doubtful cases
or classconfu-sion %
Damagescould not
bedetectedthrough
RSimagery %
Table 3. Overall ndings .
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
18/27
4.2 Variations by Communes
For Port-au-Prince, there were around 92,423 buildings which were virtually assessed. Of the to-
tal field samples collected for Port-au-Prince, 2073 virtual assessments matched with the cor-
responding field sample, and the overall accuracy was found to be 56%. Figure 17 charts the
performance accuracy of the user and the producer, which dips sharply in the Moderate class.
Kappa statistics was around 25.29%. The accuracy level went up to 28% when the error matrix
was applied with just two classe s. Furthermore, the postmortem of th e errors showed th at around20% of them could have been avoided by the analyst and around 9% occurred due to practical
impossibilities with the aerial images.
Figure 16. Showing kappa statistics for various communes.
33
Of 290,672 buildings virtually assessed, we had a field sample of 1.5 %, or 4329 buildings. The
accuracy of the joint database was found to be 61%. The Moderate Damage class was the most
poorly performed class in the entire assessment, with the Kappa statistics around 23%. The accu-
racy level increased by 25% when the error matrix was applied with just two classes. Furthermore,
the postmortem of the errors showed that around 15-20% of errors could have been avoided by
the analyst and that around 10% obtained because of practical impossibilities with aerial images
as applied to building damages. Further analysis on the accuracy levels for each organization
showed that UNOSAT performed better in the overall accuracy, followed by World Bank first and
then EC-JRC. For the Kappa statistical analysis, EC-JRC performed slightly better the other two
organizations.
Figure 17. Showing kappa statistics for various agencies
Figure 15. Showing comparison on four and two class for various agencies.
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
19/27
4.3 Four-Class and Two-Class Variations
It would be interesting to see how the accuracy levels behaved if the entire damage assessment
was carried out with just two classes, Destroyed and Not Destroyed. It was evident from the start
that the four-class approach was an ambitious and time consuming undertaking. If the accuracy
is better for two classes than four classes, then it would be wise to start with two damage classes
immediately after the disaster and then later improvise with more classes. Figure 18 is a hypotheti-
cal scenario to study the accuracy assessment with two classes; slight adjustments were made in
combining the EMS-98 scale.
4.4 Post-mortem of Failed Class
The question confronting us is how the results of the accuracy assessment can lead to a construc-
tive discussion of ways to improve the reliability of the numbers..It is important to discover why it
failed and what are the chances of improving it.
From the previous sections we know that there are limitations with the aerial images and that the
outcome of the photo interpretation method is heavily dependent on analyst performance. Fig-
ure 19 shows the three most likely factors producing the failed results. They are:
1.Building damages which could not be detected through remote sensing imagery, the
main limitation of using the aerial images.
2.Errors which could have been avoided by the analyst.
3.Doubtful cases or class confusion.
If we could assess postmortem the failed classes in a more refined manner, researchers would
have a better way of gauging the potential accuracy of aerial imagery assessment for disasters
of similar scale in the future. The failed assessment points need to be analysed individually with
the pre-and post-images. This by itself is a time consuming exercise if the accuracy levels are very
low.
FOUR CLASS % TWO CLASS %
PETION VILLE
ALL COMMUNES CROIX DES BOUQUES
DELIMASCIRE SOLIEL
TABRRE LEOGANE
CARREFOUR
GRESSIR
PORT-AU-PRINCE
56
5667
76
82
61
61
35
44
46
100
90
80
70
60
50
40
30
20
10
94
92
89
74
84
78
72
63
94
86
35Figure 18. Showing accuracy levels comparison between four and two class.
Figure 19. Showing logic behind post mortem of failed class.
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
20/27
DISCUSSION5
5.1 Methodology
Binomial distribution was used for the sample size calculation, and the majority of the research-
ers are using the same for error matrix or the normal approximation of the binomial distribution to
compute required sample size. This seems to be a standard practice. According to Congalton,
a general rule of thumb is a minimum of 50 samples for each category in error matrix; our sam-
ples meet that requirement. But it was unfortunate that we could not have samples collected for
Jacmel, Petit Goave, and Gran Goave communes where EC-JRC was responsible for the asse ss-
ment.
Another interesting as pect of the sample is the use of lan d cover maps with land use data as one
of the input parameters. The land use map is a gross approximation based on land cover. The
use of the virtual building damage assessment as an input criteria could be debatable depend-
ing on whether it will influence the sample selection. But considering the use of this at the grid
level to ensure the sample is spread evenly on the study area is not a bad idea in and of itself
The knowledge and expertise of the field sample collectors in using the EMS-98 scale is question-
able even though UNOSAT and EC-JRC had briefed them in detail about the scale. Had structural
engineers or other experts collected the field sample, they would have had different results. But
we have to admit that CNIGS were the best availabl e choice gi ven their local knowledge of the
situation.
The accuracy of the GIS and the other manual errors associated with the field sample collection
is an issue to be discussed. From the field data, around 300 samples were discarded during pre-
processing either because they were marked on the streets or away from the building footprints or
because there were two samples with varying class over the same building footprint. Even though
this exercise was tedious, it was necessary for quality assurance on the field sample and, in addi-
tion, personally rechecking the samples improves the confidence level of the data. Among other
practical issues tied to the CNIGS side of the data collection are a lack of access to certain grids
due to nonexistent roads and relatively high slopes, difficulty differentiating one building from
another even on a map 1/1000, and problems sending photos due to low speed Internet con-
nections.
As we saw in the in troduction, more than 10 agencies were involv ed in the v irtual da mage as-
sessment for Haiti, and when I looked closely at their maps, I realized that there was no similarity in
defining the damage in the Legends. It might be impossible for a person in the field to compare
these maps and to achieve common understanding of the damages based on the damage
classes. There could be a need for closer inspection here to determine whether it is possible to
develop a standardized catalogue in rapid mapping of virtual damage assessment situations.
In conjunction with this, we saw that it is more difficult to coordinate viable postmortem analyses
with common field data produced by different agencies relying on different interpretations of
damage classes.
37
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
21/27
5.2 Overall Communes
The error matrix has become widely accepted as the standard method for reporting the accuracy
of the GIS data derived from the remote sensing data. The analysis in the previous section very
clearly indicates accuracy levels fluctuating between 35% and 82%. This wide range of band in
the accuracy makes it difficult to draw direct conclusions from the results. The overall accuracy
from the error matrix for all the communes put together for all four classes was aroun d 60% . This
rate was much improved by 25 % with just two classes. This is a valuable lesson as far as the dam-
age classes are concerned. Directly after the disaster, it would be wiser to confine the study to just
two classes and then later expand it to more classes if required.
We know t hat Kappa statistics is a measure of agreement rather than the percent age of occu r-
rence. Landis and Koch have provided instructions for interpreting K values (Table 3). This is not a
proven frequency but more of a guideline. The average of the Kappa statistics for all the com-
munes put together is around 21%, just barely falling into the category of fair agreement. This
result raises questions about the strength of the photo interpretation method since almost all the
agency values are around 20%. The perspective of the individual analyst assigned to damage
assessment, myself included, is important to any analysis of the reasons for the errors. My experi-
ence indicates that many of the problems derived from ambiguous building definitions. The pro-
cess would have gone more smoothly if building definitions had been agreed upon before the
start of the exercise.
KAPPA VALUES INTERPRETATION
< 0 No agreement
0.0 0.20 Slight agreement
0.21 0.40 Fair agreement
0.41 0.60 Moderate agreement
0.61 0.80 Substantial agreement
0.81 1.00 Almost perfect agreement
As I mentioned in the introd uction, without spatial data o n the property boundaries and land use
data, it is a complicated situation to define a building. I will provide some specific examples.
In the case of a building apparently not affected but in the midst of a destroyed area, analysts
struggled to choose between the moderate and severely destroyed classes. There were also
situations where buildings were marked as destroyed when in fact they were under construction.
Closer attention to the pre- and post- images was required to ascertain the status of the building.
Depending on the length of time between the pre-scenario and the post-scenario, there was a
chance the house had been completed during this gap, thus creating a wrong entry as under
construction. In some instances, the presence of elements like internally displaced camps in the
streets or road blocks gave a false impression that the buildings around it had been affected by
the earthquake. But we realized that people were moving out of their houses into the streets fear-
ing aftershocks, which was another reason the analyst might make a wrong decision on the as-
sessments. Shantytowns or urban slums posed a significant challenge to the analyst owing to the
dense nature of the buildings and their poor quality of construction with unconventional building
materials.
Table 4. Range on Kappa Statistics Agreement
39
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
22/27
CONCLUSION & RECOMMENDATIONS6
CONCLUSION & RECOMMENDATIONS
This study offers some valuable lessons for future damages assessment analysis. The data reveal
the limitations of aerial images and especially photo interpretation method in building damage
assessments. The reliability of virtual damage assessment through photo interpretation is put intoquestion and leads us to ponder the best method going forward. On a more detailed level, the
study elaborated the specific errors to which may be attributed the low percentage of accuracy.
The four-class damage categorization could have been avoided and the whole damage as-
sessment instead progressed from the start with two classes. This would have allowed the analyst
more time, and many of the class confusion errors could have been avoided as well. If the initial
digitalization of the building points (without error assessment) had been out sourced completely
via crowd sourcing or with academic institutes who were willing to participat e, overall results could
have been improved, as happened in the case of the Delmas area of Port-au-Prince. The field
data collection could have been better coordinated to avoid repetition of the preprocessing of
the field data both by CNIGS and UNOSAT. (For example, building points were moved by CNIGS
over the building footprints after the field survey, but they were not snapped over the virtual as-
sessment points; this work was carried out by the author.) For UNOSAT, the labour shifts might have
been better organized, and a detailed training catalogue would have been welcomed by theanalysts.
Time constraints and the unavoidable confusion attendant upon a disaster of this magnitude
precluded analysts from exploiting the full potential of such a study. While much can be gleaned
from the analysis, circumstances did not allow other desired avenues of investigation: more ex-
tensive review of the existing literature; samples collected from the Jacmel, Petit-Goave, and
Grand Goave communes; a comparison of accuracy levels of satellite imagery of damages and
aerial imagery; refined procedures for sample collection and sample size distribution.
My research points to the importance of standardizing the building damage catalogue. This could
possibly require a detailed research and intervention in the rapid mapping community. Research
in this field should also look to alternative methods of damage assessments using high resolution
imagery like object-based image analysis.
41
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
23/27
BIBLIOGRAPHY7
1. Canadian International Development Agency. (2005). Haiti: facts at a glance. .
2. CIA (2010). The Central Intelligence Agency World Factbook. .
3. Congalton R. G. 1991. A review of assessing the accuracy of classifications of remotely sensed
Data. Remote Sensing of Environment.
4. Darcy, J. & Hofmann, C. (2003). Needs assessment and decision-making in the humanitarian
sector. London: Humanitarian Policy Group, Overseas Development Institute.
5. Eberhard, M. O., Baldridge, S., Marshall, J., Mooney, W., & Rix, G. J. (2010). USGS_EERI_
HAITI_V1.1.pdf (application/pdf Object). Web. 00/00/10.
6. Ernest, H. Preeg (1992). The Haitian Dilemma. International Studies.
7. Geospatial Data Availability for Haiti: An Aid in the Development of GIS-Based Natural
Resource Assessments for Conservation Planning.
8. Haiti_Reconnais sance.pdf. (2010). http://www.eqclearinghouse .org/20100112- haiti/wp-con-
tent/uploads/2010/02/Haiti_Reconnaissance.pdf.
9. Handbook for Estimating the Socio-economic and Environmental Effects of Disasters. (2003).
ECLAC & World Bank http://www.gdrc.org/uem/disasters/disenvi/VOLUME%20I.pdf. Wiley. ISBN
0-471-15227-7.
10. Lillesand, T.M, Kiefer, R.W., and Chipman, J. W. (2003). R emote sensing and image
interpretation (5th ed.).
11. Manuel Roig-Franzia(2010). Washington post, retrieved from http://www.washingtonpost.com/
wp-yn/content/articl e/2010/01/19/AR2010011904614.html
12. Rao, (2000) Disaster management
13. UN (2010). United Nations Office for the Coordination of Humanitarian Affairs. Haiti Earthquake
Situation Report #21, February 16, 2010.
14. UNOSAT.(2010).Atlas of building damage asse ssment, http://unosat .web.cern.ch/unosat/
shared/Haiti-HQ-2010/maps/PDNA/PDNA_atlas_intro.pdf.
43
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
24/27
ANNEXURE7
Annexure 1
Satellite Tasking Satellite Sensor Time (UTC) Area Type
1 SPOT 5 14&19 Jan Port-au-Prin ce
2 BJ-1 SensonML (red band /
infrared band) 13-Jan Port-au-Prin ce
3 ALOS AVNIR-2 13-Jan
PALSAR 16-Jan
AVNIR-2 23-Jan
PRISM 23-Jan
PAN 23-Jan
4 DigitalGlobe multiple dates
WV-01 13&14 Jan multi ple
collection dates Mapping entire region B/W
WV-02 14 Jan multiple
collection dates Port-au-Prin ce Multi
spectral
5 QuickBird multiple collection
dates Port-au-Prince
6 Ikonos 14&15&17 Jan
7 GeoEye-1 13&16&18 Jan
8 Cosmo-Skymed 15-Jan
9 Formosat-2 13&14&15&16 &17
Jan Port-au-Prince
10 Radarsat-2 14&15 Jan Port-au-Prin ce
11 HJ-1-A/B CCD, Hyper-spectra l
camera, Infrared
camera 14-Jan
12 RapidEye 13&14&15& 17
Jan full country
coverage
13 EO-1 15-Jan Port-au-Prince ALI
14 EROS-B 17-Jan Port-au-Prin ce
45
Source: http://www.un-spider.org/haiti
Satellite tasking after the earthquake.
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
25/27
Annexure 2
Negligible to sligbt damage(no Structual damage) Hair-line cracks in very ferw wallsfall of smallpleces of laasteronly. Fall of loose stones fromupper parts of buildings invery frw class
Moderate damage(slight structural damage.moderate non- structuraldamage) Cracks in manuwalls fall of fairly large piecesof plaster. parts of chlmmysfall down
Substantial to heavydamage (moderate
structural damage, heavynon- structural damage)
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
26/27
CROIX -DES- BOUQUES
TABARRE
CITE-SOLIEL
PORT-AU-PRINCE
DELMAS
Annexure 4
49
Four class and two class accuracy.
-
7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti
27/27
Institute of International Health, Immunology & Microbiology
Faculty of Health Sciences, University of Copenhagen