Visual Information Visual Information SystemsSystems

Lilian TangLilian Tang

Description of Content Description of Content – image processing– image processing

Primitive image propertiesPrimitive image properties Through image processing techniquesThrough image processing techniques Colour, texture, local shapeColour, texture, local shape The need of combination of these properties The need of combination of these properties

into a consistent set of localised propertiesinto a consistent set of localised properties There can be weighting scheme to balance the There can be weighting scheme to balance the

importance of each type of property. importance of each type of property. Image features Image features

ColorColor One is that the recorded color varies One is that the recorded color varies

considerably with the orientation of the considerably with the orientation of the surface, the viewpoint of the camera, the surface, the viewpoint of the camera, the position of the illumination, the spectrum position of the illumination, the spectrum of the illuminant, and the way the light of the illuminant, and the way the light interacts with the object. interacts with the object.

Second, the human perception of color is Second, the human perception of color is an intricate topic where many attempts an intricate topic where many attempts have been made to capture perceptual have been made to capture perceptual similarity. similarity.

ColourColour

TextureTexture• Texture is a phenomenon that is Texture is a phenomenon that is

widespread, easy to recognise and widespread, easy to recognise and hard to define. hard to define.

TextureTexture• Together with colour, texture is a Together with colour, texture is a

powerful discriminating feature, powerful discriminating feature, present almost everywhere in nature. present almost everywhere in nature.

• Like colours, textures are connected Like colours, textures are connected with psychological effects. In with psychological effects. In particular, they emphasize particular, they emphasize orientations and spatial depth orientations and spatial depth between overlapping object.between overlapping object.

Traditional Definition of Traditional Definition of TextureTexture

Texture refers to a spatially repeating pattern on Texture refers to a spatially repeating pattern on a surface that can be sensed visuallya surface that can be sensed visually

In the image, the apparent size, shape, spacing In the image, the apparent size, shape, spacing etc, of the texture elements (the texels) do etc, of the texture elements (the texels) do indeed varyindeed vary Varying distances of the different texels from the Varying distances of the different texels from the

cameracamera Varying foreshortening of the different texels. Varying foreshortening of the different texels.

texture gradientstexture gradients - systematic change in the - systematic change in the size and shape of the elements making up a size and shape of the elements making up a texturetexture

recover shape from texture

Recover Shape From TextureRecover Shape From Texture After some mathematical analysis , one After some mathematical analysis , one

can compute expressions for the rate of can compute expressions for the rate of change of various image texel features, change of various image texel features, such as area, foreshortening, and density. such as area, foreshortening, and density. These texture gradients are functions of These texture gradients are functions of the surface shape as well as its slant and the surface shape as well as its slant and tilt with respect to the viewer.tilt with respect to the viewer.

To recover shape from texture, one can To recover shape from texture, one can use two-step process: use two-step process: 1) measure the texture gradients 1) measure the texture gradients 2) estimate the surface shape, slant, and 2) estimate the surface shape, slant, and

tilt that would give rise to the measured tilt that would give rise to the measured texture gradients.texture gradients.

RecentRecent Technical Definition of texture

Texture is a broad term used in pattern recognition to identify image patches (of any size) that are characterized by differences in brightness.

• Techniques to extract meaningful texture descriptors from image are many, based on different models and assumptions.

• An effective representation of textures can be based on statistical and structural properties of brightness patterns.

Texture Content MeasurementTexture Content Measurement• Textures may be described according to Textures may be described according to

their spatial, frequency or perceptual their spatial, frequency or perceptual properties. Periodicity, coarseness, properties. Periodicity, coarseness, preferred direction, degree of complexity preferred direction, degree of complexity are some of the most perceptually salient are some of the most perceptually salient attributes of a texture. attributes of a texture.

• Feature spaces based on these attributes Feature spaces based on these attributes are particularly interesting for image are particularly interesting for image retrieval by texture similarity.retrieval by texture similarity.

Space – based modelsSpace – based models Auto-correlation functionAuto-correlation function A texture can A texture can

be represented taking into account the be represented taking into account the spatial size of grey-level primitives. Fine spatial size of grey-level primitives. Fine textures have a small size of their grey-textures have a small size of their grey-level primitives. Coarse textures a large level primitives. Coarse textures a large size.  size.  

Co – occurrence matrixCo – occurrence matrix A different way of A different way of measuring textures is by taking into measuring textures is by taking into account the spatial arrangement of grey-account the spatial arrangement of grey-level primitives.level primitives.

Calculate TextureCalculate Texture

energy

entropy

contrast

Integration of primitive Integration of primitive propertiesproperties

a separation between color, local a separation between color, local geometry, and texture. geometry, and texture.

an integrated view on color, texture, and an integrated view on color, texture, and local geometry is urgently needed as only local geometry is urgently needed as only an integrated view on local properties can an integrated view on local properties can provide the means to distinguish among provide the means to distinguish among hundreds of thousands different images. hundreds of thousands different images.

Further research is needed in the design of Further research is needed in the design of complete sets of image properties with complete sets of image properties with well-described variant conditions which well-described variant conditions which they are capable of handling. they are capable of handling.

Image featuresImage features

Image featuresImage features Grouping DataGrouping Data, , Global and Accumulating Global and Accumulating

FeaturesFeatures, , Salient Features, SignsSalient Features, Signs, , Shape and Shape and Object FeaturesObject Features, , Description of Structure and Description of Structure and Lay-Out Lay-Out

Also in the description of the image by features, Also in the description of the image by features, it should be kept in mind that for retrieval a total it should be kept in mind that for retrieval a total understanding of the image is rarely needed.understanding of the image is rarely needed.

the deeper one goes into the semantics of the the deeper one goes into the semantics of the pictures, the deeper the understanding of the pictures, the deeper the understanding of the picture will also have to be picture will also have to be 

With segmentationWith segmentation no segmentation no segmentation 

Interpretation And Similarity Interpretation And Similarity MeasureMeasure

Semantic features aim at encoding Semantic features aim at encoding interpretations of the image which may be interpretations of the image which may be relevant to the application.relevant to the application.

feature set can be explained feature set can be explained derives an unilateral interpretation from the derives an unilateral interpretation from the

feature setfeature set compares the feature set with the elements in a compares the feature set with the elements in a

given data set on the basis of a similarity functiongiven data set on the basis of a similarity function In content-based retrieval, it is useful to push In content-based retrieval, it is useful to push

the semantic interpretation of features the semantic interpretation of features derived from the image as far as one can. derived from the image as far as one can.

Similarity MeasurementSimilarity Measurement A different road to assigning a A different road to assigning a

meaning to an observed feature set, meaning to an observed feature set, is to compare a pair of observations is to compare a pair of observations by a similarity function. – a kind of by a similarity function. – a kind of interpretationinterpretation And this is the advantage to have And this is the advantage to have

content-based retrieval.content-based retrieval.

Colour Histogram Based Colour Histogram Based RetrievalRetrieval

This method is to retrieve images from the This method is to retrieve images from the database that have perceptually similar database that have perceptually similar colour to the input image or input description colour to the input image or input description from the user.from the user.

The basic idea is to quantize each of the RGB The basic idea is to quantize each of the RGB values into m intervals resulting in a total values into m intervals resulting in a total number of mnumber of m33 colour combinations (or bins) colour combinations (or bins)

A colour histogram H(I) is then constructed. A colour histogram H(I) is then constructed. This colour histogram is a vector {h1, h2, …, This colour histogram is a vector {h1, h2, …, hmhm33} where element hx represents the } where element hx represents the number of pixels in image I falling within bin number of pixels in image I falling within bin x.x.

Colour Histogram Based Colour Histogram Based RetrievalRetrieval

The colour histogram becomes the index of this imageThe colour histogram becomes the index of this image To retrieve image from the database, the user supplies To retrieve image from the database, the user supplies

either a sample image or a specification for the system either a sample image or a specification for the system to construct a colour histrogram H(Q).to construct a colour histrogram H(Q).

A distance metric is used to measure the similarity A distance metric is used to measure the similarity between H(Q) and H(I). Where I represents each of the between H(Q) and H(I). Where I represents each of the images in the database. And example distance metric images in the database. And example distance metric is shown as follows:is shown as follows:

x=mx=m33

D(Q, I ) = D(Q, I ) = |q |qxx-i-ixx|| x=1x=1

Where qWhere qx x and iand ixx are the numbers of pixels in the image are the numbers of pixels in the image Q and I, respectively, falling within bin x.Q and I, respectively, falling within bin x.

Colour Histogram Based Colour Histogram Based RetrievalRetrieval

may fail in recognizing images with may fail in recognizing images with perceptually similar colours but no common perceptually similar colours but no common colours. This may be due to a shift in colour colours. This may be due to a shift in colour values, noise or change in illumination. – values, noise or change in illumination. – measure the similaritymeasure the similarity

Not enough for complicated images where Not enough for complicated images where spatial position is more important information. spatial position is more important information.

May combine with other methods such as May combine with other methods such as shape and /or texture based retrieval to shape and /or texture based retrieval to improve the accuracy of the retrieval.improve the accuracy of the retrieval.

Semantic SimilaritySemantic Similarity knowledge-based type abstraction hierarchiesknowledge-based type abstraction hierarchies concept-space concept-space a linguistic description of texture patch visual a linguistic description of texture patch visual

qualities is given and ordered in a hierarchy of qualities is given and ordered in a hierarchy of perceptual importance on the basis of perceptual importance on the basis of extensive psychological experimentation. extensive psychological experimentation.

A more general concept of similarity is needed A more general concept of similarity is needed for relevance feedback, in which similarity with for relevance feedback, in which similarity with respect to an ensemble of images is required. respect to an ensemble of images is required.

Different Levels of Content-base Different Levels of Content-base Indexing and RetrievalIndexing and Retrieval

syntactical level: mainly deal with colour, shape, syntactical level: mainly deal with colour, shape, texture etc. Some used manual annotation to texture etc. Some used manual annotation to index dataindex data e.g. retrieval system let users to fill forms to provide e.g. retrieval system let users to fill forms to provide

queries, like location, colour etc categories, like the queries, like location, colour etc categories, like the work done in Berkeleywork done in Berkeley

semantic levelsemantic level: : analyse captionsanalyse captions purely used text information and didn’t make use of purely used text information and didn’t make use of

the information inherent in the imagesthe information inherent in the images complicated algorithm applied on small scalecomplicated algorithm applied on small scale

Multi-level indexingMulti-level indexing An advantage of image indexing based An advantage of image indexing based

on multi-level contents rather than on multi-level contents rather than solely on low-level features such as solely on low-level features such as texture and colours, is that it would texture and colours, is that it would readily provide the basic framework readily provide the basic framework required for "semantic interoperability" required for "semantic interoperability" when one tries to search through, not when one tries to search through, not only one, but a federation of image only one, but a federation of image collections from different disciplines.collections from different disciplines.

Learning from FeedbackLearning from Feedback The interacting user brings about many new The interacting user brings about many new

challenges for the response time of the system.challenges for the response time of the system. Content-based image retrieval is only scalable to Content-based image retrieval is only scalable to

large data sets when the database is able to large data sets when the database is able to anticipate what interactive queries will be made.anticipate what interactive queries will be made.

A frequent assumption is that the image set, the A frequent assumption is that the image set, the features, and the similarity function are known in features, and the similarity function are known in advance. In a truly interactive session, the advance. In a truly interactive session, the assumptions are no longer valid. assumptions are no longer valid.

A change from static to dynamic indexing is A change from static to dynamic indexing is required.required. (Arnold 2000) (Arnold 2000)

An integrated issueAn integrated issue It will demand its own view of things It will demand its own view of things

as it is our belief that content-based as it is our belief that content-based retrieval in the end will not be part of retrieval in the end will not be part of the field of computer vision alone. the field of computer vision alone. The man-machine interface, domain The man-machine interface, domain knowledge, and database technology knowledge, and database technology each will have their impact on the each will have their impact on the product.product.

Arnold 2000

SummarySummary• The heritage of computer vision. The heritage of computer vision. • The influence on computer vision. The influence on computer vision.

• Deal with large data sets. Deal with large data sets. • the absence of a general method for strong the absence of a general method for strong

segmentation. segmentation. • has revitalized interest in color image processing.has revitalized interest in color image processing.• attention for invariance has been revitalized  attention for invariance has been revitalized  

• Similarity and learning. Similarity and learning. • Interaction. Interaction. • The need for databases. The need for databases. • The problem of evaluationThe problem of evaluation. . • The semantic gap and other sources. The semantic gap and other sources.