1

Integrating User Feedback Log into Relevance Feedback by Coupled SVM for

Content-Based Image Retrieval

9-April, 2005

Steven C. H. Hoi * , Michael R. Lyu *, Rong Jin #

*Department of Computer Science & EngineeringThe Chinese University of Hong Kong

Shatin, N.T., Hong Kong SAR

# Department of Computer Science and EngineeringMichigan State University

East Lansing, MI 48824, USA

The 1st IEEE EMMA Workshop in conjunction with 21st IEEE ICDE, Japan, April, 2005.

2

Outline• Introduction• Background

Log-based Relevance Feedback

• Coupled Support Vector Machine Support Vector Machine Formulation Alternating Optimization A Practical Algorithm

• Experimental Results• Conclusion

3

Introduction

• Content-based Image Retrieval (CBIR) An important component in visual information retrieval QBE: query-by-example based on low-level visual

features Semantic gap: low-level features, high-level concepts

QBE

4

Introduction

• Relevance Feedback (RF) A powerful tool to attack the semantic gap problem Interactive mechanism to solicit users’ feedbacks Boost the retrieval performance of CBIR greatly Many existing techniques already…

• Problems Regular relevance feedback needs too many rounds of i

nteractions for achieving satisfactory results.

5

Introduction• Motivation

Can user feedback log be used to improve the Can user feedback log be used to improve the regular relevance feedback?regular relevance feedback?

Relevance Feedback

User Feedback Log

?

Problem

6

Background• Log-based Relevance Feedback (LRF)

Relevance Matrix: R RF round / Log session: Nl images are marked Elements: relevant (1), irrelevant (-1), unknown (0)

Log Sessions

Image samples

1 -1 1 -1 -1 0 1 -1 -1 11

-1 1 -1 -1 -1 -1 -1 1 -1-10

7

Background

• Learning Problem for LRF Low-level image content:

User feedback log:

Multi-Modal Learning Problem

},,,{ 21 NxxxX

},,,{ 21 NrrrR

8

Coupled Support Vector Machine

• Motivation How to attack the learning problem on the two modalities?

• Low-level Image content: X

• User relevance feedback log: R Support Vector Machines: superior classification performance

• A Straightforward Solution: Learn an SVM classifier on each modality respectively

• For image content X, we learn an optimal weighting vector w;

• For log content R, we learn an optimal weighting vector u; Combine their results together linearly

9

• A Straightforward Solution For the image content modality: wTx

For the user feedback log modality: uTr

Coupled Support Vector Machine

10

• Disadvantages of the straightforward solution Linear combination Modality Consistence

• Our better solution: Coupled SVM Learn the two modalities in a unified formulation Enforce the prediction on the two types of information

to be consistent.

Coupled Support Vector Machine

11

• Formulation: Coupled SVM

Coupled Support Vector Machine

12

• Optimization of Coupled SVM Hard to be solved directly Alternating Optimization (AO)

• AO: two-step optimization Fix Y’, try to find (u, b_u), and (w, b_w) Fix (u, b_u) and (w, b_w), try to find Y’

Coupled Support Vector Machine

13

• Alternating Optimization Fix Y’, the primal optimization is equivalent to solving

the two optimization subproblems:

Coupled Support Vector Machine

14

• Alternating Optimization (AO) By introducing non-negative Lagrange multipliers, the

above two subproblems can be solved

Coupled Support Vector Machine

15

• Alternating Optimization (AO) After solving (u, b_u) and (w, b_w), fixing them, the op

timal Y’ can be found to fit the data as follows:

Coupled Support Vector Machine

16

• Summary of AO procedure 1) Beginning with a small value of 2) Performing the two-step AO procedure 3) Repeating 2) by increasing until it achieves the setting

threshold

• Comments on the Coupled SVM Can be a general approach for multi-modal learning problems Need to investigate the convergence issue of Alternating

Optimization Need to study better methods for solving the optimization

problem Require to take some practical considerations when fitting for

specific problems.

Coupled Support Vector Machine

17

• A Practical Algorithm Practical considerations

• Cannot engage all unlabeled samples due to response requirement for relevance feedback

• Strategy for choosing unlabeled samples– Closest to the decision boundary of SVM: most informative

according to active learning

– Closest to the labeled samples: to avoid too much effort in learning the label information

• Introducing a parameter to control the error for label correction to avoid overlarge change in the labeled set

Coupled Support Vector Machine

18

• A Practical Algorithm (cont’d)

Coupled Support Vector Machine

19

• A Practical Algorithm (cont’d)Coupled Support Vector Machine

20

Experimental Results

• Dataset Images selected from COREL image CDs Two ground-truth datasets

• 20-Category: each category contains 100 images, totally 2,000

• 50-Category: each category contains 100 images, totally 5,000

21

Experimental Results (cont’d)

• Low-level Image Representation Color Moment

• 9-dimension

Edge Direction Histogram • 18-dimension

• Canny detector, 18 bins of 20 degrees each

Wavelet-based texture • 9-dimension

• Daubechies-4 wavelet, 3-level DWT

• Entropies of 9 subimages are generated for the texture feature

22

Experimental Results (cont’d)

• Collection of User Log Data Log format

• A log session (LS) corresponds a relevance feedback round

• Each log session contains 20 images labeled by users

Log data• On 20-Category: 161 log sessions

• On 50-Category: 184 log sessions

23

Experimental Results (cont’d)• CBIR GUI for collecting feedback data

24

Experimental Results (cont’d)• Performance Evaluation

Measurement Metric• Average Precision = # relevant images / # returned images

Experimental Setting• 100 queries• 20 initially labeled images• SVM: RBF kernel, parameters set via training data

Comparison Schemes• RF-SVM

– traditional relevance feedback by SVM

• LRF-2SVM– log-based relevance feedback by learning two SVMs respectively

• LRF-CSVM– log-based relevance feedback by Coupled SVM

25

Experimental Results (cont’d)• Performance Evaluation: on 20-Category Dataset

26

Experimental Results (cont’d)• Performance Evaluation: on 50-Category Dataset

27

Experimental Results (cont’d)

28

Experimental Results (cont’d)

29

Conclusion• A log-based relevance feedback scheme was studied by

integrating user feedback log into the content learning of low-level visual features in content-based image retrieval.

• A general multimodal learning technique, i.e. Coupled Support Vector Machine, was proposed for studying the data with multiple modalities.

• A practical algorithm by Coupled SVM was presented to attack the log-based relevance feedback problem in CBIR.

• Experimental results show our proposed scheme is effective for the log-based relevance feedback problem.

30

Q&A

31

References• Chu-Hong Hoi and Michael R. Lyu, A Novel Log-based Relevance Fe

edback Technique in Content-based Image Retrieval, in Proc. ACM Multimedia, New York, USA, 10-16 October, pp. 24-31, 2004

• S. Tong and E. Chang. Support vector machine active learning for image retrieval. In Proc. ACM Multimedia, pages 107--118, 2001.