Chapter 14 Chapter 14 Accuracy AssessmentAccuracy AssessmentAccuracy AssessmentAccuracy Assessment

Introduction to Remote Sensing, Introduction to Remote Sensing, James B. Campbell James B. Campbell pp

O tlineO tlineOutlineOutline

DefinisiDefinisiAccuracy & PrecissionAccuracy & PrecissionSignificanceSignificanceSignificanceSignificance

Source of Classification ErrorSource of Classification ErrorError CharacteristicsError CharacteristicsMeasurement of Map AccuracyMeasurement of Map Accuracy

Error MatrixError MatrixOmission & CommissionOmission & CommissionUser & Producer AccuracyUser & Producer Accuracy

Interpretation of the Error matrixInterpretation of the Error matrixPercentage CorrectPercentage CorrectPercentage CorrectPercentage CorrectQuantitative Assessment of Error matrixQuantitative Assessment of Error matrix

DefinitionDefinitionDefinitionDefinition

Accuracy : correctness, mengukur “kecocokan” antara Accuracy : correctness, mengukur “kecocokan” antara suatu image yg tidak diketahui kualitasnya dengan suatu image yg tidak diketahui kualitasnya dengan sebuah standar imagesebuah standar imagePrecission : detail, “The distinction is important because Precission : detail, “The distinction is important because one may be able to increase accuracy by decreasing one may be able to increase accuracy by decreasing precission”, precission”, Meningkatkan detail = menambah ragam kategori . Misal Meningkatkan detail = menambah ragam kategori . Misal : forest = caniferous, pine, shortleaf pine atau mature : forest = caniferous, pine, shortleaf pine atau mature shortleaf pineshortleaf pine akan menambah peluang klasifikasi akan menambah peluang klasifikasi errorerrorerror error Statistical context : high accuracy = low bias, (estimated Statistical context : high accuracy = low bias, (estimated value is consistenrly close to an accepted reference value is consistenrly close to an accepted reference value)value)value) value)

DefinitionDefinitionDefinitionDefinition

DefinitionDefinitionDefinitionDefinition

SignificanceSignificanceAccuracy has many practical implication :Accuracy has many practical implication :Accuracy has many practical implication : Accuracy has many practical implication : effect legal standing, operational usefulness, effect legal standing, operational usefulness, validity for scientific research.validity for scientific research.

So ce of Classification E oSo ce of Classification E oSource of Classification ErrorSource of Classification Error

Manual Interpretation : Misidentification, Manual Interpretation : Misidentification, Excessive generalization, Error registration, Excessive generalization, Error registration, Variation in detail of interpretation etc.Variation in detail of interpretation etc.Character of landscape : parcel size, variation in Character of landscape : parcel size, variation in

l i l id i i b fl i l id i i b fparcel size, parcel identities number of parcel size, parcel identities number of categories, arrangement of categories, number categories, arrangement of categories, number of parcel per category shapes of parcelof parcel per category shapes of parcelof parcel per category, shapes of parcel, of parcel per category, shapes of parcel, radiometric and spectral contrast with radiometric and spectral contrast with surrounding parcelsurrounding parcelsurrounding parcelsurrounding parcel

So ce of Classification E oSo ce of Classification E oSource of Classification ErrorSource of Classification Error

Three error types dominate: Three error types dominate: Data Acquisition Errors: These include sensor performance, stability of Data Acquisition Errors: These include sensor performance, stability of the platform, and conditions of viewing. We can reduce them or the platform, and conditions of viewing. We can reduce them or compensate for them by making systematic corrections (e g bycompensate for them by making systematic corrections (e g bycompensate for them by making systematic corrections (e.g., by compensate for them by making systematic corrections (e.g., by calibrating detector response with oncalibrating detector response with on--board light sources generating board light sources generating known radiances). We can make corrections, often modified by known radiances). We can make corrections, often modified by ancillary data such as known atmospheric conditions, during the initial ancillary data such as known atmospheric conditions, during the initial processing of the raw data. processing of the raw data. p gp gData Processing Errors: An example is misregistration of equivalent Data Processing Errors: An example is misregistration of equivalent pixels in the different bands of the Landsat Thematic Mapper. The goal pixels in the different bands of the Landsat Thematic Mapper. The goal in geometric correction is to hold the mismatch to a displacement of no in geometric correction is to hold the mismatch to a displacement of no more than one pixel. Under ideal conditions, and with as many as 25 more than one pixel. Under ideal conditions, and with as many as 25

d l (GC ) d d l hd l (GC ) d d l hground control points (GCP) spread around a scene, we can realize this ground control points (GCP) spread around a scene, we can realize this goal. Misregistrations of several pixels significantly compromise goal. Misregistrations of several pixels significantly compromise accuracy. accuracy. SceneScene--dependent Errors: As alluded to in the previous page, one such dependent Errors: As alluded to in the previous page, one such error relates to how we define and establish the class which in turn iserror relates to how we define and establish the class which in turn iserror relates to how we define and establish the class, which, in turn, is error relates to how we define and establish the class, which, in turn, is sensitive to the resolution of the observing system and the reference sensitive to the resolution of the observing system and the reference map or photo. Mixed pixels fall into this category. map or photo. Mixed pixels fall into this category.

So ce of Classification E oSo ce of Classification E oSource of Classification ErrorSource of Classification Error

E o Cha acte isticsE o Cha acte isticsError CharacteristicsError Characteristics

Classification error : assignment pixel to one category Classification error : assignment pixel to one category that different from true category ( as determined that different from true category ( as determined ground observation/groundground observation/ground--truth ).truth ).

Error Characteristic :Error Characteristic :Error are not distributed over the image at random, display a Error are not distributed over the image at random, display a g , p yg , p ydegree of systematic, ordered occurrence in space.degree of systematic, ordered occurrence in space.Often erroneously assigned pixels are not spatially isolated but Often erroneously assigned pixels are not spatially isolated but occur grouped in areas of varied size and shape (Campbell 1981)occur grouped in areas of varied size and shape (Campbell 1981)Errors may have specific spatial relationships to the parcels toErrors may have specific spatial relationships to the parcels toErrors may have specific spatial relationships to the parcels to Errors may have specific spatial relationships to the parcels to which they pertain, for example, they may tend to occur at which they pertain, for example, they may tend to occur at edges or in the interiors of the parcelsedges or in the interiors of the parcels

E o Cha acte isticsE o Cha acte isticsError CharacteristicsError Characteristics

Tiga macam eror patern dari Landsat, Cogalton (1984). Dark area = error clasification, white area = correct.

Meas ement of Map Acc acMeas ement of Map Acc acMeasurement of Map AccuracyMeasurement of Map Accuracy

Compare the “true map”/ reference map, Compare the “true map”/ reference map, (asumsi lebih akurat) with image to be (asumsi lebih akurat) with image to be ( ) g( ) gevaluated.evaluated.Jika pembandingan tanpa memperhatikanJika pembandingan tanpa memperhatikanJika pembandingan tanpa memperhatikan Jika pembandingan tanpa memperhatikan posisi pixel, klasifikasi total bisa dianggap posisi pixel, klasifikasi total bisa dianggap sama meskipun sebenarnya posisi dengansama meskipun sebenarnya posisi dengansama meskipun sebenarnya posisi dengan sama meskipun sebenarnya posisi dengan image reference tidak sesuaiimage reference tidak sesuai sitesite--spesific accuracyspesific accuracyspesific accuracyspesific accuracy

Meas ement of Map Acc acMeas ement of Map Acc acMeasurement of Map AccuracyMeasurement of Map Accuracy

Site & Non Site Specific ErrorSite & Non Site Specific Error

Meas ement of Map Acc acMeas ement of Map Acc acMeasurement of Map AccuracyMeasurement of Map Accuracy

Error Matrix : matrik perbandingan image Error Matrix : matrik perbandingan image reference dengan image yang akan reference dengan image yang akan g g y gg g y gdianalisa berdasarkan kelompok klasifikasi dianalisa berdasarkan kelompok klasifikasi pixelpixel--pixel yang sama dalam imagepixel yang sama dalam image--image image pp p y g gp y g g ggtersebut.tersebut. dari Error Matrik dapat dari Error Matrik dapat dihitung % correct,dihitung % correct,g ,g ,

% correct = sum agreement pixel between reff & image(jumlah % correct = sum agreement pixel between reff & image(jumlah diagoal pada error matrix)/total pixeldiagoal pada error matrix)/total pixel

Meas ement of Map Acc acMeas ement of Map Acc acMeasurement of Map AccuracyMeasurement of Map Accuracy

Error MatrixError Matrix

Meas ement of Map Acc acMeas ement of Map Acc acMeasurement of Map AccuracyMeasurement of Map Accuracy

Compiling Error matrixCompiling Error matrixImage direpresentasikan dengan pixel2Image direpresentasikan dengan pixel2Image direpresentasikan dengan pixel2Image direpresentasikan dengan pixel2Hitung jumlah pixel untuk tiap klasifikasiHitung jumlah pixel untuk tiap klasifikasiYg perlu diperhatikan : Klasifikasi referenceYg perlu diperhatikan : Klasifikasi referenceYg perlu diperhatikan : Klasifikasi reference Yg perlu diperhatikan : Klasifikasi reference dengan image yg akan diklasifikasi harus dengan image yg akan diklasifikasi harus compatiblecompatible turunan klasifikasi harus masih turunan klasifikasi harus masih ppsesuai dengan kategori pada referencesesuai dengan kategori pada reference

Meas ement of Map Acc acMeas ement of Map Acc acMeasurement of Map AccuracyMeasurement of Map Accuracy

Meas ement of Map Acc acMeas ement of Map Acc acMeasurement of Map AccuracyMeasurement of Map Accuracy

Omission & Commission ErrorOmission & Commission ErrorOmission : jumlah pixel pada reference image yang tidak sesuai Omission : jumlah pixel pada reference image yang tidak sesuai dengan kategori kalsifkasi pada image yg dievaluasidengan kategori kalsifkasi pada image yg dievaluasig g p g ygg g p g ygCommission : jumlah pixel pada image yang dievaluasi yang Commission : jumlah pixel pada image yang dievaluasi yang tidak sesuai dengan keadaan sebanarnya/klasifkasi pada tidak sesuai dengan keadaan sebanarnya/klasifkasi pada referencereference

CA (Customer Accuracy) & PA (Produsen Accuracy)CA (Customer Accuracy) & PA (Produsen Accuracy)CA : jumlah pixel pada image yg dievaluasi, yang sesuai dengan CA : jumlah pixel pada image yg dievaluasi, yang sesuai dengan kondisi reference dibandingkan jumlah total pixel pada image yg kondisi reference dibandingkan jumlah total pixel pada image yg d l k kl f k bd l k kl f k bdievaluasi untuk klasifikasi tsb.dievaluasi untuk klasifikasi tsb.PA : jumlah pixel pada image yg dievaluasi, yang sesuai dengan PA : jumlah pixel pada image yg dievaluasi, yang sesuai dengan kondisi reference dibandingkan jumlah total pixel pada image kondisi reference dibandingkan jumlah total pixel pada image referencereferencereference.reference.

Meas ement of Map Acc acMeas ement of Map Acc acMeasurement of Map AccuracyMeasurement of Map Accuracy

http://rst.gsfc.nasa.gov/Sect13/Sect13_3.html

Interpretation of the Error Interpretation of the Error ppmatrixmatrix

Percentage Correct (PC)Percentage Correct (PC)Ukuran yg sering dipakaiUkuran yg sering dipakaiBeberapa rekomendasi :Beberapa rekomendasi :

PC = 85 % dibutuhkan untuk landPC = 85 % dibutuhkan untuk land--use data resource use data resource management (Anderson et al, 1976)management (Anderson et al, 1976)management (Anderson et al, 1976)management (Anderson et al, 1976)FitzpatrickFitzpatrick--Lins(1978) akurasi dari USGS landLins(1978) akurasi dari USGS land--cover map cover map untuk central Atlantic coastal : 85 % (untuk skala 1:24.000), untuk central Atlantic coastal : 85 % (untuk skala 1:24.000), 77% (1:100.000), 73% (1:250.000)77% (1:100.000), 73% (1:250.000)77% (1:100.000), 73% (1:250.000)77% (1:100.000), 73% (1:250.000)Untuk automasiUntuk automasi--interpretasi dari Landuse menggunakan interpretasi dari Landuse menggunakan hanya data MSS PC yang didapat = 38%, dan untuk MSS + hanya data MSS PC yang didapat = 38%, dan untuk MSS + ancillary data PC = 78 % (Tom et al. 1978)ancillary data PC = 78 % (Tom et al. 1978)ancillary data PC 78 % (Tom et al. 1978) ancillary data PC 78 % (Tom et al. 1978)

http://rst.gsfc.nasa.gov/Sect13/Sect13_3.html

I t t ti f th E t iI t t ti f th E t iInterpretation of the Error matrixInterpretation of the Error matrix

Quantitative Assessment of the Error Matrix Quantitative Assessment of the Error Matrix kappa (k) = measured of difference between kappa (k) = measured of difference between observed observed

tt b t t db t t d th t th tth t th tagreementagreement between two map and between two map and the agreement that the agreement that might be attained solely by chance matching two mapmight be attained solely by chance matching two map..

k = (observed k = (observed ––expected)/(1expected)/(1-- expected) expected) Observed = percentage correctObserved = percentage correctExpected = product row & column,Expected = product row & column, change agreement change agreement two categories when two images superimposedtwo categories when two images superimposed (Fig (Fig 14 8)14 8)14.8)14.8)

I t t ti f th E t iI t t ti f th E t iInterpretation of the Error matrixInterpretation of the Error matrix

I t t ti f th E t iI t t ti f th E t iInterpretation of the Error matrixInterpretation of the Error matrix

k = 0 83k = 0 83 accuracy = 83% better thanaccuracy = 83% better thank 0.83 k 0.83 accuracy 83% better than accuracy 83% better than expected from chance assignment of pixel expected from chance assignment of pixel to cattegoriesto cattegoriesto cattegories.to cattegories.k = +1,k = +1, accuracy = 100%, perfect accuracy = 100%, perfect classification table 14 6classification table 14 6classification, table 14.6 classification, table 14.6

Pen t pPen t pPenutupPenutup

Accuracy dibutuhkan sebagai ukuran informasi yang didapatkan Accuracy dibutuhkan sebagai ukuran informasi yang didapatkan mendekati nilai standar/referensi tertentu/nilai sebenarnyamendekati nilai standar/referensi tertentu/nilai sebenarnya

Untuk aplikasi tertentu direkomendasikan menggunakan suatu nilaiUntuk aplikasi tertentu direkomendasikan menggunakan suatu nilaiUntuk aplikasi tertentu direkomendasikan menggunakan suatu nilai Untuk aplikasi tertentu direkomendasikan menggunakan suatu nilai accuracy tertentu. Selain itu accracy juga berdampak pada nilai accuracy tertentu. Selain itu accracy juga berdampak pada nilai legal dari data dan informasi yang dihasilkan.legal dari data dan informasi yang dihasilkan.

Accuracy didapatkan dengan membandingkan dengan suatu image Accuracy didapatkan dengan membandingkan dengan suatu image referensi tertentu, yg dianggap benar, lebih akurat dstreferensi tertentu, yg dianggap benar, lebih akurat dst

Untuk mengukur accuracy digunakan alat bantu error matrixUntuk mengukur accuracy digunakan alat bantu error matrixUntuk mengukur accuracy digunakan alat bantu error matrix, Untuk mengukur accuracy digunakan alat bantu error matrix, dengan menghitung percentage correct, omission&comission error, dengan menghitung percentage correct, omission&comission error, PA & CA dan kappa, semuanya untuk melihat kerelatifan kebenaran PA & CA dan kappa, semuanya untuk melihat kerelatifan kebenaran klasifkasi yang telah dilakukan.klasifkasi yang telah dilakukan.