Image Reconstruction for Quality Assessment of Edge Detectors

Barghavi Govindarajan*†

*Dept. of Electrical Engineering Tufts University, Medford, USA

†BA Logix, Inc., Quincy, USA bgovin01@eecs.tufts.edu

Karen A. Panetta*†

*Dept. of Electrical Engineering Tufts University, Medford, USA

†BA Logix, Inc., Quincy, USA karen@eecs.tufts.edu

Sos Agaian‡

‡Dept. of Electrical Engineering University of Texas at San Antonio

San Antonio, USA sos.agaian@utsa.edu

Abstract—Extraction of the edges is a key step in image processing and there is still a continuing research effort to develop new and effective edge detection algorithms. Despite this fact, there is no single, reliable and efficient metric to evaluate the quality of an edge detector [15]. We introduce an original method for image reconstruction that leads to edge evaluation based on image estimation [11]. A new quantitative metric for assessment of the performance of the edge detector is also presented. The operation of the measure is established on a diverse image database using standard edge detection algorithms and the one based on partial derivatives of Boolean functions [1]. The uses of the measure for an assortment of purposes are demonstrated and these are backed by visual assessment as well as some distance-based error functions applied on synthetic images.

Keywords— Edge evaluation, detector parameters, image reconstruction, weighted median, interpolation, quality measure, structural similarity

I. INTRODUCTION

Edge detection is of prime importance in a variety of applications including segmentation, recognition and enhancement for purposes ranging from security and commerce to medical diagnosis. A discontinuity in image brightness can be assumed to correspond to a discontinuity in depth, surface orientation, reflectance or illumination. The fact that edges in the image domain also constitute a strong link to physical properties of the world make it necessary to develop an accurate evaluation of the phenomena that created the image as perceived by the elusive human visual system. Unlike other common signal processing applications, such as compression or resizing/interpolation, there is no ground truth (GT) of actual edge locations typically known which makes the comparison of the achieved output to an “ideal” is impossible. Edge detector evaluation has been a challenge also because of the erratic nature of algorithms and the correlation between the existing measures and algorithms [8, 9]. An independent, general measure for the performance of the edge detector is therefore necessitated by the need for automated or intelligent vision systems, parameterization of algorithms, comparison of existing methods and detection of correlations amongst image features, the methodology, the application and the system’s limiting conditions. The authors of [17] summarize in great detail the existing image and edge quality measures and

categorize them based on their underlying concepts and fit to specific applications. However, as many researchers point out, there is not yet a global measure of the quality of an edgemap that is viable for a large range of databases or different types of edge detectors [6-7, 16, 21-23]. Here, we introduce a new method for the reconstruction of an image from an edgemap. This provides a framework for evaluating the quality of the edgemap assuming that edges hold the most important information in a signal and that the quality of a reconstructed estimation of the image based on this information reflects the quality of the output edge map [11]. We proposed in an earlier work, the reconstruction estimation as a way of determining the quality of an edgemap as well as the optimal parameters of the detector. This concept and our algorithm of reconstruction estimation for edge evaluation is described briefly in Section II. It also includes the demonstration of a practical new-fangled method of system automation. In Section III, we evaluate the performance of the measure itself by its agreement with results of subjective analysis and distance-based error metrics. Finally, a section of this paper is dedicated to showing the various purposes for which our evaluation procedure can be adopted.

Original Image Binary Edgemap

Estimator:Reconstruct

T

Edge Tube

EI

Comparator: Measure M

Edge Detector

I & E

Reconstructed Image

R

Figure 1: The edgemap evaluation scheme that uses reconstruction tion estimation and a measure M to quantify the similarity between I and R.

691

1-4244-2384-2/08/$20.00 c© 2008 IEEE

II. RECONSTRUCTION ESTIMATION Binary edge maps are the traditional output from edge

detection techniques. Their quantitative evaluation techniques can be classified into several categories: human observations, probabilistic measures, ROC (receiver operating characteristic) curve analysis, distance measures, edge connectivity and width uniformity, motion evaluation, “task-based” empirical evaluation and reconstruction estimations to mention a few. Techniques do exist [7] for generating an estimated ground truth (EGT) for use in comparison of natural images, but these suffer from their own biases regarding the algorithm selection during the generation of the EGT. Amongst these measures, the reconstruction-based measures quantitatively evaluate the resulting edge map with no knowledge of the actual edges (thus, no GT required). Fig. 1 illustrates the technique for any image, edge detector and image quality measure assuming little more than the intuitive relation between a well reconstructed image and the quality of the edge detector that created the edge map from which the image was estimated.

A. Estimator: Interpolation of Edge Pixels

The procedure used to recover the original image I(i,j) from the edgemap E(i,j), described in [11] is as follows. The original gray values I(i,j) with (i,j) k are used as interpolating values for the minimization of the (discretized version of the) functional,

didj2

+2 RR

(1)

The function R(i,j), which minimizes Eq. 1 under the constraints in Eq. 2 is the recovered image.

kj)(i,)j,i(I=)j,i(R (2)

We present a rudimentary realization of this using a linear interpolation which is depicted in Fig. 2. Note that ‘T’ is the “edge tube” as defined in [24].

Figure 2: Interpolation of Edge Pixels for reconstruction estimation (Ref. Eq.3, 4) using weighted average technique. The results are good; however, the scheme lacks robustness to noise and is incapable of preserving sharpness.

8

1=kkd

8

1=k kTkd=)j,i(R (3)

1618

3×153+2×128+3×191=)j,i(R (4)

Eq. 4 shows the interpolated value at the marked pixel in the example in Fig. 2. The use of an averaging method for linear interpolation works satisfactorily and has been used for the purposes of evaluation of image by the authors of this paper in [18]. In this work, we establish an innovative alternative to the reconstruction algorithm. Some of the other functions that replace the linear interpolation as well include polynomials (a generalization of the linear functions for interpolation), rational functions, trigonometric polynomials, splines or other statistical techniques. Here is a demonstration of the median interpolation. Consider the same example in Fig. 2. We obtain a better estimation of R(i,j) by using for its value, the weighted median of [T1,T2,…T8] where the weights are [1/d1,1/d2,…1/d8] respectively. The formula of weighted median is reiterated by Morales, Boman et al. in [19]. For the purpose of interpolation, we proceed as follows (Refer Eqs.5,6,7). Given a set of positive integers W, the output at every non-edge pixel is given by R(i,j). E(i,j) represents the input

samples and the is the replicator operator defined as Eq. 8.

}{ )j,i(k1/d=j)(i,k W;)j,i(8W),....,j,i(2W),j,i(1W=)j,i(W (5)

)]j,i(8T)j,i(8W),....,j,i(1T)j,i(1W[MED=)j,i(R (6)

),j,i(8T),...j,i(2T),j,i(1T=)j,i(T(7)

timesk

W

kT,......,kT,kT=)j,i(kT)j,i(kW (8)

The value at the marked pixel in the example would be:

128]1533/1,1282/1,1913/1[MED=)j,i(R (9)

That the new reconstruction estimation using median interpolation (REMI) is far better than simple linear interpolation is evident from Fig. 3. REMI circumvents creation of false edges during reconstruction, retains the sharpness of the edge tube, is robust to noise and produces visually pleasing results.

B. Comparator: Measure of similarity for Images

A better image estimation allows better performance of the quality measure. To compare the quality of the estimated image (R(i,j)) with the original input, we use improved SSIM . An illustrative example is provided in Fig. 4. The modified/ improved measure of structural similarity in images (SSIM) as introduced in our earlier work [18]. The expression is restated in Eq. 10.

Search in T in all 8 directions for edge pixels: [T1, T2,, T8]

Edge Tube: T

191

128

153

Get pixel: (i,j)

Evaluate distance of first edge pixel

encountered in each direction: [d1, d2,.. d8]

R(i,j) as in Eq. 3

692 2008 IEEE International Conference on Systems, Man and Cybernetics (SMC 2008)

Original Image Edge map Reconstruction: Weighted Average

Reconstruction: Weighted Median (REMI)

Figure 3: Interpolation using weighted average and weighted median on two images. Note the performance of median interpolation in spite of the noise in the edge map. Averaging (or alternately, any higher degree polynomial interpolation) causes eruption of false edges and smoothing effects.

IMPROVED SSIM = 20*[SSIM]10 (9)

where )+(×)+(

)2(×)2(=SSIM 2

y2x

2y

2x

xyyx(10)

In Eq. 10, x and y are the two input images and and are the mean and standard deviation of 4X4 windows in the image. SSIM quantifies the similarity between x and y and is ranges from 0-1 with SSIM=1 representing a perfect match.

Original Image Edge map (1) Edge map (2)

COMPARE: REMI (1) REMI (2) Improved SSIM 1.6937e-041 3.9853e-040

Figure 4: Using the REMI and improved SSIM to evaluate quality of edgemaps (1) and (2) obtained using two diferent edge detectors. The greater the SSIM, better the edgemap.

III. USE OF REMI In this section we discuss in detail the various methods in

which the REMI scheme and the improved SSIM measure can be put to use. This includes automation of edge detection (parameterization), tuning of edgemap outputs, comparison of edgemaps for real and synthetic images and establishing correlations between system features.

A. Automation/ Parameterization

To increase efficiency, productivity, quality and to optimize resources/ reduce the implementation costs, it is necessary to automate vision- based systems. Sometimes the chief problem encountered in the process is the determination of the value of parameters in the chosen algorithm. The issue is more pertinent to edge detectors as almost every algorithm has a thresholding step with a parameterized thresholding function which directly affects the quality of the edge map. Fig. 5 gives the step-wise details of the procedure to follow in order to detect the optimal value of the parameter without human intervention. A demonstration of this procedure constitutes Fig. 6a. The advantage of the method is that it works for binary as well as gray scale images. It is noteworthy that an average/ polynomial interpolation technique for reconstruction estimation would not accurately reconstruct a binary image due to the generation of non-binary values. Although the problem is circumvented by introducing an additional threshold step, REMI is a convenient, robust and accurate alternative. Fig. 7 depicts how our new edge evaluation technique is deployed in automated edge detection using the algorithm based on partial derivatives of Boolean functions (PDBF) [1, 18]. Owing to the automation, the fact

2008 IEEE International Conference on Systems, Man and Cybernetics (SMC 2008) 693

that the PDBF detector has multiple parameters improves the usability of the algorithm without degrading its implementation efficiency/ demanding more resource allocations. We mentioned earlier about the quality of edgemap being better with an increased SSIM. In the context of parameter selection, one needs to account for the fact that very noisy edgemaps lead to a well populated edgetube. Hence the REMI yields an image of very good quality which will be reflected by the improved SSIM too. Empirically, we observed that improved SSIM values close to 0.3 (and not as high as 20) gave highly usable edgemaps that were also pleasing to view. The coupling of the discrete differential plot to the automation algorithm agrees with the intuition that we need to zero in on that value of the parameter that produces “stable” edgemaps. In other words, the parameter value of interest to us outputs only the most critical information that do not change by tiny fluctuations in the value. At this point, the slope of the differential curve is close to zero and closest to the decreasing improved SSIM curve.

Original Image

(a) =1.0 (b) =1.4 (c) =1.8

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-3

-2

-1

0

1

2

3

4 x 10-38

parameter values

mea

sure

s

ALPHA Profile

20*MSSIM10

diff(20*MSSIM10)

Figure 6: “Alpha Profile” of the original image shown above. The point of least non-zero distance between the two curves in the plot corresponds to the best possible value of parameter that yields a good edge map.

B. Comparison of Edge Detectors

The improved SSIM has a better dynamic range than the structural similarity metric in Eq. 10. This allows it to distinguish images with greater accuracy. As was highlighted in Fig. 4, the value of the improved SSIM is directly related to the quality of the output. The results of comparison of edgemaps from the PDBF edge detector and some standard edge detection algorithms are presented in a subsequent section.

C. Discover Correlations

Since the quality-assessment scheme described here does not assume anything about the underlying principle of edge detection or feature of edgemaps that are evaluated, the measurement technique is viable across a range of applications, images and edge detectors. Creating a parameter versus quality curve (“parameter profile”) predictably unearths significant correlations between the image category and the value assumed by the parameter, the application and the quality of expected output etc. Table 1 gives the reader an idea about the kind of relations that were noticed in our diverse test database consisting of 4 synthetic and 22 natural images classified based on the applications such as medicine, security or satellite or texture imagery.

D. Tune algorithm

The measure is a handy tool for tuning the algorithm as per the requirements of the system. This is to avoid getting edgemaps with more details than what is required.

REMI

DP= Min Value; N=0

Input Image, Min/ Max/ Step of Detector parameter (DP)

Calculate/ Store IMP_SSIM=

Improved SSIM (Eq. 9)

DP = Min + N*Step; N=N+1

DP>Max

Detector: Obtain Edge map

NY

Calculate DIFF_SSIM=

)SSIM_IMP()DP(d

d

Calculate DIST_SSIM = IMP_SSIM - DIFF_SSIM

Optimal DP = DP(x DIST_SSIM(x) 0)

Figure 5: Automated Parameter Selection (APS)– Procedure for determiningthe optimal value of a detector parameter

a

b

c

694 2008 IEEE International Conference on Systems, Man and Cybernetics (SMC 2008)

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-10

-5

0

5

10

15

20

parameter values

measures

MSSIM20*MSSIM10

diff(20*MSSIM10)

Original Image Polynomial based binarization (obtained using the progressive technique described in [18])

Improved PDBF

FUSION + THRESHOLD (Eq. 11)

=1.75

Final Edge map

Improved PDBF Improved PDBF Improved PDBF

IV. COMPUTER SIMULATIONSFig. 6 gives an overview of the automated edge detector

system. For demonstration purposes we use the algorithm that employs PDBF [1, 18]. The use of a specific thresholding algorithm also influences the performance of edge detector. The threshold methodology used here is the ALPHA threshold method. For an MXN image, for every block B(i,j) located at pixel, (i,j), the edges are marked by

elseMN

MNI9

ijBif

01

j,iFα

>= (11)

The PDBF algorithm has been modified by us to yield improved edgemaps. The binary images (obtained from binarization described in [18]) are fed into the partial derivatives edge detector wherein the input is processed using a windowing technique [1]. Every 2X2 block within the image is extracted and represented in a Boolean functional form and the partial derivative with respect to each Boolean variable calculated. Subsequently, we fuse the partial derivatives-images by a logical OR operator as against classical real addition. This makes the algorithm highly efficient as the threshold processing block at the front end of the system has been removed. It is our observation that the resulting

edgemaps are less noisy and hence better. Based on the experiments conducted, Table 2A-B has been created. The former shows that the improved SSIM measure backs the results presented in our previous work that was based on visual inspection. The latter compares visually and automatically chosen parameter values.

Table 1: Sample relations that were discovered during analysis of edgemaps with the REMI & improved SSIM scheme

Operation Discovery

Tuning of parameter ALPHA

(ref.. Eq 11)

Using lower values of alpha in the earlier stages in the algorithm reduced number of binarized images progressively input to the detector (ref. [18])

Parameter Selection

Security applications involving detection of arms require 3<ALPHA<5 in order to dump the right objects sought by the recognition system. The parameter value has to do with the intensity values of the material involved.

Quality Assessment of edgemaps for various

ALPHA

In our medical image database, an improved SSIM of only 0.2 corresponded to a well usable result with neatly segmented image parts. This was particularly true for tumor applications where the aim is to segment the area and not so much as to detect every intensity edge in the surrounding tissues.

Figure 6: The automated PDBF Edge Detector performs well

2008 IEEE International Conference on Systems, Man and Cybernetics (SMC 2008) 695

Table 2A: The improved SSIM (ISSIM) concurs with FOM (synthetic images)

Image

Canny

ISSIM:FOM: 07595

6.6498e-011

FOM: 0.9165

3.5857e-027

FOM: 0.5738

3.3996e-006

FOM: 0.8909

1.5559e-023

LoG

ISSIM:FOM: 0.8291

3.5599e-010

FOM: 0.9239

3.6386e-027

FOM: 0.2841

1.3447e-031

FOM: 0.8839

5.1940e-024

PDBF

ISSIM: FOM: 0.7502

6.2830e-011

FOM: 0.9597

8.6856e-026

FOM: 0.9565

4.3287e-003

FOM: 0.8989

5.9342e-023

Table 2B: The improved SSIM (ISSIM) concurs with manual selection Image (Visual Inspection) (REMI)

I 2.5 2.3

II 1.2 1.2

III 3.5 3.5

IV 1.5 1.4

In Table 2, Pratt’s Figure of Merit is, F is evaluated by

{ } ( )= +=

AN

kAI kdNNF

121

1,max

1α

(12)

where NI is the number of actual edge pixels (from the GT) and NA is the number of detected edge pixels. The distance parameter d(k) is the distance from the kth actual edge to the corresponding detected edge [15]. From the values of hand-picked values of Alpha (based on visual appeal) and automatically selected parameter values for a partial list of images from our database, the error percentage in alpha is approximately 12% which is an acceptable and very low figure.

V. CONCLUSIONS The PDBF detector was proven to discover the optimal

edges and is marked by good edge revelation, high fidelity as well as localization capabilities. The issue of determining the large number of adjustable parameters, which can affect the computation time and effectiveness of the algorithm, was solved by the novel reconstruction estimation-based edge evaluation scheme that renders automatic parameter selection

capability to the detector. REMI is a quantitative evaluation scheme that is both automated and effective, while remaining viable for a large range of applications. The scheme could also serve to improve the performance of traditional edge detector.

ACKNOWLEDGMENT

This research is supported in part by BA Logix, Inc., The authors of this paper wish to thank Dr. Ethan Danahy for his helpful suggestions.

REFERENCES

1. E. Danahy, K. Panetta, S. Agaian, “Feature Extraction System for Contextual Classification within Security Imaging Applications,” IEEE Sys. of Sys. Eng., April 2007.

2. F. Pellegrino, W. Vanzella, and V. Torre, “Edge Detection Revisited,” IEEE Trans. Sys., Man, and Cyber. – Part B., Vol. 34, No. 3, June 2004.

3. M. Basu, “Gaussian-Based Edge-Detection Methods – A Survey,” IEEE Trans. Systems, Man, and Cyber. – Part C, Vol. 32, No. 3, Aug. 2002.

4. Z. Rahman and D. Jobson, “Noise, edge extraction and visibility of features,” SPIE Defense and Security Symp.: Visual Info. Processing XIV, Vol. 5817, p. 200-211, March 2005.

5. S. Agaian, J. Astola, and K. Egiazarian, Binary Polynomial Transforms and Nonlinear Digital Filters, Marcel Dekker, Inc., New York, 1995.

6. M. Heath, S. Sarkar, T. Sanocki, and K. Bowyer, “Comparison of Edge Detectors: A Methodology and Initial Study,” Comp. Vision and Image Understanding, Vol. 69, No. 1, p. 38-54, 1998.

7. Y. Yitzhaky and E. Peli, “A Method for Objective Edge Detection Evaluation and Detector Parameter Selection,” IEEE Trans. Pattern Ana. Mach. Intell. Vol. 25, No. 10, Oct 2003.

8. S.Baker,,“Global Measures of Coherence for Edge Detector Evaluation,”IEEE Conf. Comp. Vision, Pattern Rec.,Vol.2, p. 373, 1999.

9. L. Forbes, B. Draper, “Inconsistencies in Edge Detector Evaluation,” IEEE Conf. Comp. Vision Pattern Rec., Vol. 2, p. 398-404, June 2000.

10. W. Pratt, Digital Image Processing,JohnWiley&Sons, NY, 2nd ed., 1991. 11. S. Carlsson, “Sketch based coding of gray level images,” IEEE Trans.

Image Processing, Vol. 15, No. 1, p. 57-83, Jan. 1988. 12. J. Elder, “Are Edges Incomplete?” International Journal of Computer

Vision, Vol. 34, No. 2-3, p. 97-122, August 1999. 13. T. Nguyen and D. Ziou, “Contextual and non-contextual performance

evaluation of edge detectors,” Pattern Rec. Let., No. 21, p. 805-16, 2000. 14. T. Kanungo, M. Jaisimha, et. al., “A Methodology for Quantitative

Performance Evaluation of Detection Algorithms,” IEEE Trans. Image Processing, Vol. 4, No. 12, Dec. 1995.

15. Avcibas, I., Sankur, B., and Sayood, K. (2002), “Statistical evaluation of image quality measures”, Journal of Electronic Imaging, 11:206—23

16. N. L. Fernandez, et.al., “Characterization of empirical discrepancy evaluation measures”, Pattern Rec. Letters, Vol. 25 (1), Jan. 2004.

17. I. Knowles and R. Wallace, “A Variational Method for Numerical Differentiation”, Numer. Math., 70 (1995), pp. 91--110.

18. B. Govindarajan, K. Panetta, and S.Agaian, "Progressive Edge Detection on Multi-bit Images Using Polynomial-based Binarizations," Sub.: IEEE Int. Conf. Mach. Learning and Cyber., Kunming, China, July 2008.

19. Morales, A.; Boman, E.; Sung Jea Ko, "An efficient algorithm to calculate sample and rank selection probabilities for weighted median filters," Signal Processing Letters, IEEE , vol.9(5), pp.148-50, May 2002

20. J. Canny, ‘‘A computational approach to edge detection,’’ IEEE Trans., Pattern Anal. Mach. Intell. pp. 679–698, 1986.

21. Van der Heyden, F., "Evaluation of edge detection algorithms," Image Processing and its Applications, 1989., Third International Conference on , vol., no., pp.618-622, 18-20 Jul 1989

22. T. Lindeberg, “Edge detection and ridge detection with automatic scale selection”, Proc. CVPR'96, pp.465-- 470, 1996

23. Valverde, F.L, et al., "An evaluation criterion for edge detection techniques in noisy images," Int. Conf. Image Process., pp.766-69, 2001

24. T. F. Chan and J. Shen, “Mathematical Models for Local Nontexture Inpaintings”, SIAM J. Appl. Math., 62(3):1019-1043, 2001

696 2008 IEEE International Conference on Systems, Man and Cybernetics (SMC 2008)