Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf ·...
Transcript of Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf ·...
![Page 1: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/1.jpg)
Opponent Colors Revisited
Sabine Süsstrunk Image and Visual Representation Lab
![Page 2: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/2.jpg)
A small exercise…
•Ques0on: what color is [255,0,0]?
![Page 3: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/3.jpg)
A small exercise…
•Ques0on: what color is [255,0,0]? • Answer: red
S.K. Shevell
![Page 4: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/4.jpg)
Isaac Newton (1666)
http://www.newton.ac.uk/art/portrait.html
![Page 5: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/5.jpg)
https://upload.wikimedia.org/wikipedia/commons/b/bd/Dispersive_Prism_Illustration.jpg
Color is a property intrinsic to light
![Page 6: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/6.jpg)
Johann Wolfgang von Göthe…
https://en.wikipedia.org/wiki/File:Goethe_%28Stieler_1828%29.jpg
![Page 7: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/7.jpg)
“Along with the rest of the world I was convinced that all the colours are contained in the light; no one had ever told me anything different, and I had never found the least cause to doubt it, because I
had no further interest in the subject.”
https://en.wikipedia.org/wiki/Theory_of_Colours
![Page 8: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/8.jpg)
!
“But how I was astonished, as I looked at a white wall through the prism, that it stayed white! That only where it came upon some darkened area, it showed some colour, then at last, around the
window sill all the colours shone... It didn't take long before I knew here was something significant about colour to be brought forth, and I spoke as through an insGnct out loud, that the Newtonian teachings
were false.” !
Göthe
https://en.wikipedia.org/wiki/Theory_of_Colours
![Page 9: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/9.jpg)
http
s://e
n.w
ikip
edia
.org
/wik
i/The
ory_
of_C
olou
rs
Göthe, 1810
![Page 10: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/10.jpg)
Newtonian World
![Page 11: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/11.jpg)
Trichroma0c Color Vision
Young-Helmholtz!~1850
![Page 12: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/12.jpg)
Addi0ve Color Spaces/Encodings
Visible (Red, Green, and Blue: RGB)
(s)RGB!XYZ!LMS
![Page 13: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/13.jpg)
Opponent colors
•BeUer correla0on with color appearance and color discrimina4on data. -‐ Göthe 1810; Hering 1872; Munsell 1910, Jameson & Hurvich 1955, 1957.
Hering, 1872 Munsell, 1920http
://w
ww
.han
dprin
t.com
/HP/
WCL
/col
or2.
htm
l
http
://m
unse
llcol
or.w
ebno
de.p
t/m
unse
ll-co
lor-t
ree/
![Page 14: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/14.jpg)
Recep0ve Fields in the Re0na
Kolb, 2003
R.L. DeValois, C.J. Smith, S.T. Kitai, and A.J. Karoly. Responses of single cells in different layers of the primate lateral geniculate nucleus to monochromatic light. Science, 127:238–239, 1958.
![Page 15: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/15.jpg)
Opponent Colors
• Spectrally non-‐selec4ve recep0ve fields: white-‐black • Spectrally selec4ve recep0ve fields: red-‐green, blue-‐yellow
http
://w
ebvi
sion.
med
.uta
h.ed
u/
![Page 16: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/16.jpg)
Stockm
an & Sharpe, 200
2Luminance-‐chrominance
•Efficient encoding of L,M,S cone responses: -‐ De-‐correla0on of LMS cone responses result in opponent basis func0ons.
PCA
G. Buchsbaum and A. Gottschalk, Trichromacy, Opponent Colours Coding and Optimum Colour Information Transmission in the Retina. Proc. of the Royal Society of London. Series B, Biological Sciences, 1983.
![Page 17: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/17.jpg)
Opponent Color Spaces/Encoding
•Opponency works on cone contrast, not on quantum catches of cones. -‐ Different ways of modeling the contrast •Weber-‐type contrast •Log, gamma func0on, polynomial
•De-‐correla4ng transforma0on of cone fundamentals: -‐ Luma (lightness, brightness, luminance) f [L’+M’ (+S’)]: αR’ + βG’ + γB’ -‐ Red-‐green f [L’-‐M’]: αR’ -‐ βG’ (+ γB’) -‐ Blue-‐yellow f [S’ -‐ (L’+M’)]: γB’ -‐ (αR’ + βG’)
•Euclidian distances have some perceptual meaning.
![Page 18: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/18.jpg)
CIELAB Color Space/Encoding
• An opponent color encoding based on the MacAdam ellipses*
http
s://e
n.w
ikip
edia
.org
/wik
i/Mac
Adam
_elli
pse
*just no0ceable differences of chroma0city
reference test
![Page 19: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/19.jpg)
CIELAB Color Space
![Page 20: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/20.jpg)
Perceptual Color Difference
!
!
!
!
• Equal ΔE implies equally perceived color difference. !
!
• is considered a good approxima0on of JND (Just no0ceable difference) in simultaneous matching.
�E⇤ab =
⇥(�L⇤)2 + (�a⇤)2 + (�b⇤)2
⇤1/2
�E⇤ab ⇠ 1
![Page 21: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/21.jpg)
Salient Region Detec4on
Radhakrishna Achanta, Sheila Hemami, Francisco Estrada and Sabine Süsstrunk, Frequency-tuned Salient Region Detection, IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
![Page 22: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/22.jpg)
Salient regions...
• ...are the regions in the image that stand out (are visually conspicuous) with respect to the background.
![Page 23: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/23.jpg)
Task-‐independent Saliency
Low-‐level vision, guides human visual aUen0on
![Page 24: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/24.jpg)
Task-‐specific Saliency
http://www.facedetection.com/facedetection/misc.htm
![Page 25: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/25.jpg)
Task-‐specific Saliency
• curr
Object-detectionhttp://www.facedetection.com/facedetection/misc.htm
![Page 26: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/26.jpg)
Salient region detector
![Page 27: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/27.jpg)
Harel et al. 2006
Io et al. 1998
Ma & Zhang 2003
Hou & Zhang 2007
Walther& Koch 2006
Bian & Zhang 2009
HybridBiological-‐model based
Purely computa0onal
Saliency Detectors
ours
Center-‐surround contrast
![Page 28: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/28.jpg)
Salient Region Detectors
FS#1% FS#2%
I"#et#al.,IT98# Ma#and#Zhang,##MA03# Harel#et#al.,GB06#
Mancas#et#al.,#GR07# Bruce#and#Tsotsos,#AIM07# Hou#and#Zhang,#SR07# Achanta#et#al.,#AC08#
Zhang#et#al.,#SUN08# Bian#and#Zhang,#SWQ09#
Input#
![Page 29: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/29.jpg)
What should a good detector do?
• Emphasize the largest salient objects in an image. • Uniformly highlight whole salient regions. • Establish well-‐defined boundaries of salient objects
• Disregard high frequencies arising from texture, noise and blocking ar0facts.
• Efficiently output full resolu0on saliency maps.
![Page 30: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/30.jpg)
Saliency using Differences of Gaussians
Saliency = Abs [Center Gaussian - Surround Gaussian]
![Page 31: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/31.jpg)
center sigma axis
surround sigma axis
![Page 32: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/32.jpg)
Algorithm
•Difference of Gaussian (DoG): !
!
!
!
!
• Adding up DoGs:N�1�
n=0
G(x, y, �n+1⇥)�G(x, y, �n⇥)
= G(x, y, �N⇥)�G(x, y, ⇥)
DoG(x, y) = 12�
�1
⇥21e� (x2+y2)
2�21 � 1
⇥22e� (x2+y2)
2�22
⇥
= G(x, y, �1)�G(x, y,�2)
![Page 33: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/33.jpg)
Image Average
Gaussian Blur
Iµ = [Lµ, aµ, bµ]
Ilp = [Llp, alp, blp]
S(x, y) = ||Iµ � Ilp||
Our Saliency Detec0on Algorithm
![Page 34: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/34.jpg)
Image Average
Gaussian Blur
Iµ = [Lµ, aµ, bµ]
Ilp = [Llp, alp, blp]
S(x, y) = ||Iµ � Ilp||�EOur Saliency Detec0on Algorithm
![Page 35: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/35.jpg)
Image Segmenta0onCompute saliency map
Mean-‐shiv segmenta0on
Compute average saliency for the
image Sµ
For any segment k, compute avg. saliency Sk
If Sk > 2 x Sµ choose segment (binarize)
Input image
Compare with ground truth!
Extract object
![Page 36: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/36.jpg)
Sample segmenta0on results
![Page 37: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/37.jpg)
Image Segmenta0on Results
Itti et al.
ours
Ma & Zhang
Hou & Zhang
Harel et al.
Achanta et al.
Based on an accurate object-‐bound ground truth of 1‘000 images
![Page 38: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/38.jpg)
Improve Edge Detec0on
![Page 39: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/39.jpg)
Document Correc0on
![Page 40: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/40.jpg)
Combine low-‐level and task-‐specific Saliency
![Page 41: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/41.jpg)
Image Resizing
Original Seam carving Seam carving with saliency
RK. Achanta and S. Süsstrunk, Saliency Detection for Content-aware Image Resizing, IEEE ICIP, 2009.S. Avidan, and A. Shamir, Seam Carving for Content-Aware Image Resizing, ACM SIGGRAPH, 2007.
Videos
![Page 42: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/42.jpg)
Comparisons
Original Seam carving Seam carving with saliency
RK. Achanta and S. Süsstrunk, Saliency Detection for Content-aware Image Resizing, IEEE ICIP, 2009.S. Avidan, and A. Shamir, Seam Carving for Content-Aware Image Resizing, ACM SIGGRAPH, 2007.
Videos
![Page 43: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/43.jpg)
Radhakrishna Achanta, Appu Shaji, Pascal Fua, and Sabine Süsstrunk, Image Summaries using Database Saliency, ACM SIGGRAPH ASIA, 2012.
![Page 44: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/44.jpg)
Superpixels and Supervoxels
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk, SLIC Superpixels Compared to State-of-the-Art Superpixel Methods, IEEE Transaction of Pattern Analysis and Machine Intelligence, 2012.
![Page 45: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/45.jpg)
Superpixels
• Superpixels are pixel clusters of an over-‐segmented image.
![Page 46: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/46.jpg)
State-‐of-‐the-‐Art
• Graph based algorithms -‐ [GS04] Efficient graph based segmenta0on [Felzenszwalb and HuUenlocher, IJCV 2004] -‐ [NC05] Normalized cuts [Mori, ICCV 2005] -‐ [SL08] Superpixel laoce [Moore et al. CVPR 2008] -‐ [GC10] Graph cuts [Veksler et al. ECCV 2010]
• Gradient based algorithms -‐ [WS91] Watershed [Vincent and Soille, PAMI 1991] -‐ [MS02] Mean shiv [Comaniciu and Meer, PAMI 2002] -‐ [TP09] Quick shiv [Vedaldi and SoaUo, ECCV 2008] -‐ [QS09] Turbopixels [Levinstein et al. PAMI 2009]
![Page 47: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/47.jpg)
Input GS04 NC05Felzenszwalb and HuUenlocher G. Mori
QS09 TP09Levinstein et al.Vedaldi and SoaUo
SLIC
State-‐of-‐the-‐Art
![Page 48: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/48.jpg)
48
Input GS04 NC05Felzenszwalb and HuUenlocher G. Mori
QS09 TP09 SLICLevinstein et al.Vedaldi and SoaUo
Good boundary adherence but no control over the number of superpixels
![Page 49: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/49.jpg)
49
Input GS04 NC05Felzenszwalb and HuUenlocher G. Mori
QS09 TP09 SLICLevinstein et al.Vedaldi and SoaUo
Control over the number of superpixels but poor boundary adherence
![Page 50: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/50.jpg)
50
Input GS04 NC05Felzenszwalb and HuUenlocher G. Mori
QS09 TP09 SLICLevinstein et al.Vedaldi and SoaUo
Control and good boundary adherence
![Page 51: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/51.jpg)
SLIC Superpixel/voxel Algorithm
• SLIC: Simple linear itera4ve clustering • k-‐means in 3-‐6D space of color value and spa0al coordinates.
• Normalize color and spa0al distances using maximum distances within a cluster.
D =
r1
S
[(xj � xi)2 + (yj � yi)2 + (zj � zi)2] +1
m
[(Lj � Li)2 + (aj � ai)2 + (bj � bi)2]
Spatial proximity term CIELAB proximity term
![Page 52: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/52.jpg)
Choose k seeds at steps S
![Page 53: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/53.jpg)
Aver 1 itera0on
![Page 54: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/54.jpg)
Aver 10 itera0ons
2S
![Page 55: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/55.jpg)
Results
![Page 56: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/56.jpg)
Object class recogni0on
Gould et al., IJCV 2008, STAIR Vision Library
![Page 57: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/57.jpg)
Classifica0on accuracy
74.6% 75.9% 76.9%75.1% 62.0%
GS04 NC05 TP09 QS09 SLIC
Compared to NC05, SLIC is 400 0mes faster.
![Page 58: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/58.jpg)
Superpixel Graph-‐based Segmenta0on
Our result
![Page 59: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/59.jpg)
Input GS04 TP09 QS09 SLIC
Mitochondria Detec0on
![Page 60: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/60.jpg)
Supervoxels
Gökhan Yildirim, Appu Shaji, and Sabine Süsstrunk, Estimating Beauty Ratings of Videos using Supervoxels ACM Multimedia, 2013.
![Page 61: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/61.jpg)
A Lucchi, K Smith, R Achanta, G Knott, P Fua, Supervoxel-based segmentation of mitochondria in EM image stacks with learned shape features, IEEE Transactions on Medical Imaging, 2012.
![Page 62: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/62.jpg)
“Opponency” and Near-‐Infrared
M. Brown and S. Süsstrunk, Multispectral SIFT for Scene Category Recognition, Proc. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
![Page 63: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/63.jpg)
Mul0ple Spectra
400 nm 700 nm 1100 nm
Visible Spectrum Near-Infrared
Visible (Red, Green, and Blue: RGB) Near-infrared (NIR)
![Page 64: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/64.jpg)
Image Database (477 RGB+NIR)
• 477 image pairs for 9 scene categories:
![Page 65: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/65.jpg)
Correla0on between RGB and NIR
• 10’000 RGB-‐NIR pixel pairs, joint entropy:
M. Brown and S. Süsstrunk, Mul0-‐spectral SIFT for Scene Category Recogni0on, CVPR 2011.
(a) R-G (4.13) (b) B-G (4.26) (c) R-B (4.60)
(d) R-NIR (5.29) (e) G-NIR (5.30) (f) B-NIR (5.44)
Figure 3: Pairwise distributions of R, G, B and NIR pixelssampled from 10,000 examples in 100 images. The jointentropy of each distribution (64 bins, max entropy of 6 bits)is shown in parenthesis.
with robust estimation of a similarity motion model. Wereject image matches with less than 50 consistent featurematches (this occurred in only 6 instances). Our subjects areprimarily static scenes, although occasionally image motioncauses small differences between the RGB and NIR bands(around 50 of 477 cases), to which our recognition algo-rithms will need to be robust.
4. RGB-NIR Statistical DependenciesFigure 3 shows scatter plots of 10,000 sampled RGB-
NIR pixels from our dataset, plotted as pairwise distribu-tions in order of increasing entropy. Joint entropy is com-puted as H =
�ij �pij log2(pij), using a discretisation of
8 bins per dimension. Note that the visible colour entropies(R-G, R-B and B-G) are all less than the joint entropy ofvisible and NIR (R-I, G-I, B-I), with the largest entropy oc-curring for the spectral extremes (R-B and B-I). These plotssuggest that NIR gives significantly different informationfrom R, G and B, in the following we investigate whetherthis can be effectively used in a recognition context.
5. Multispectral SIFTIn their paper on colour SIFT descriptors [27], Van der
Sande et al. noted that using opponent colour spaces gavesignificantly better performance in object and scene recog-nition than computing descriptors in the R,G and B colourchannels directly. Opponent colours are present in humanvision, where early processing splits the input into achro-matic (luminance) and opponent (red-green, blue-yellow)parts. This can be explained in terms of efficient coding– opponent colours decorrelate the signal arriving at the
L, M and S photoreceptors [2, 21]. To extend the oppo-nent colour idea to RGB-NIR, we make use of the sameidea, decorrelating the 4-dimensional RGB-NIR colour vec-tor c = [r, g, b, i] by computing the eigenvectors of the co-variance matrix
⇥c =⇥
k
(ck �mc)(ck �mc)T = W�WT (3)
To ensure that the output remains as the input within the 4-dunit cube we apply a further linear mapping
c⇥i =1�
j |wij |⇥
j
wijcj ��
j w(�)ij�
j |wij |. (4)
This facilitates downstream processing which expects in-tensity values in the 0-1 range. Each component of theresulting colour vector c⇥ = [c⇥1, c⇥2, c⇥3, c⇥4] is thus a linear(scale and offset) transform of the decorrelated components.The raw RGB-NIR PCA components are shown in Figure 4.Similar to 3-dimensional colour, the first component is al-most achromatic, containing approximately equal amountsof R, G, B, and a slightly smaller NIR contribution. Thesecond component takes the difference of the spectral ex-tremes (NIR and blue), which from Figure 3 are the moststatistically independent. Subsequent components involvefurther spectral differences, and account for small fractionsof the overall signal energy (see Figure 6).
To form a multispectral SIFT keypoint, we first detectdifference of Gaussian extrema in the luminance compo-nent, and then form 4⇥4 histograms of gradient orientationsdi, i ⌅ {1..4} for each channel independently. Note thatsince the decorrelated bands (other than the first) consist ofcolour differences, the gradients consist of both spatial andchromatic differences (see Figure 5). We normalise eachcolour band independently, which equalises the weightingof the colour gradient signals, and concatenate to form thefinal descriptors. Since this can be high dimensional (512dimensions for RGB-NIR descriptors), we reduce dimen-sions using PCA, leading to a length nd descriptor
d = UT [d̃1, d̃2, ..., d̃n] (5)
where d̃i is the “clip-normalised” [11] SIFT descriptor inthe ith band, and U is a nd ⇥ 128n orthogonal matrix (typ-ically nd ⇤ 128).
6. Standard ModelsAs a baseline for comparison, we also test multispectral
versions of two standard models for category/scene recog-nition:
![Page 66: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/66.jpg)
Decorrela0on...
• PCA on 477 4-‐channel images (RGB +NIR):
M. Brown and S. Süsstrunk, Mul0-‐spectral SIFT for Scene Category Recogni0on, CVPR 2011.
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
(86.6%) (11.3%) (1.7%) (0.3%)
Figure 4: RGB-NIR “opponent” colours. The colouredbars show the amounts of red, green, blue and near-infraredin each PCA component. Note that the first two components(approximately achromatic and difference of blue-infrared)make up 98% of the signal energy.
Figure 5: Colour parts of the derivative filters used forMSIFT (note that there is an additional near-infrared com-ponent that is not visualised)
Figure 6: RGB-NIR opponent colour components of ascene. The colour transformation converts colour differ-ences into spatial patterns which are well characterised bylocal descriptors such as SIFT. Note that there is visibly lessenergy in the later components.
GIST We compute Gabor filters at 3 scales and 8 orienta-tions per scale. The image is first pre-filtered to nor-malise the local contrast, and the descriptors are con-catenated for each band. We use the implementationfrom Torralba [18].
HMAX Inspired by the “standard model” for recognitionin visual cortex, this consists of a hierarchy of filter-ing and max-pooling operations. HMAX descriptorsare computed independently and concatenated for eachband. We use the CNS implementation by Mutch etal. [15].
The HMAX filters were trained offline using a databaseof 10,000 images. GIST descriptors were normalised perband, which also gave improved performance versus globalnormalisation. In addition to our multispectral SIFT de-
scriptors, we also compare against the best performingcolour descriptor designs proposed by Van de Sande etal. [27, 3], including opponent-SIFT, C-SIFT and rg-SIFT.
7. DatabaseOur test database consists of 477 images distributed
in 9 categories as follows: Country(52), Field(51), For-est(53), Mountain(55), Old Building(51), Street(50), Ur-ban(58), Water(51). The images were processed using auto-matic white balancing for the RGB components, and equalweights on the RGB sensor responses for the NIR compo-nents, with standard gain control and gamma correction ap-plied.
8. Classification ExperimentsTo perform classification based on MSIFT descriptors,
we adopt the method of Boiman et al. [1]. This assumesthat the descriptors zi in image I are i.i.d. Gaussian, withthe class conditional density being approximated by thenearest-neighbour
p(I|c) =�
i
p(zi|c) ⇥�
i
N(zi,NNc(zi), �2) (6)
where NNc(zi) is the nearest neighbour of descriptor zi inclass c of the training set. We then use a Bayes classifier,choosing the class c� = arg maxc p(c|I) with equal classpriors. For the GIST and HMAX descriptors we use LinearSVMs (C-SVC) with a constant c parameter of 100.
9. Experiments (RGB-NIR dataset)We perform scene recognition on our dataset of 477 im-
ages, randomly selecting 99 images for testing (11 per cat-egory) and training using the rest. In all our experimentswe repeat using 10 trials with a randomly selected train-ing/test split, and quote the mean and standard deviation ofthe recognition rate (fraction of correct matches).
Firstly, we experiment with various colour representa-tions for each lifting algorithm. These are: l = luminance(greyscale), li = luminance + NIR, rgb = red, green andblue, rgbi = RGB + NIR, opp = opponent colour space(as used for opponent-SIFT), nopp = normalised opponentcolour space (as used for C-SIFT), lrg = luminance + nor-malised r, g (as used for rg-SIFT), pca1 = 1st RGB-NIRPCA component, pca2 = 1st and 2nd RGB-NIR PCA com-ponents, pca3 = RGB-NIR PCA components 1-3, pca4 =RGB-NIR PCA components 1-4, rnd = random 4�4 lin-ear transform (10 randomised transforms used over 10 tri-als each). The results are shown in Table 1. We name thecombination pca4 sift = MSIFT (the algorithm described insection 5).
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
(86.6%) (11.3%) (1.7%) (0.3%)
Figure 4: RGB-NIR “opponent” colours. The colouredbars show the amounts of red, green, blue and near-infraredin each PCA component. Note that the first two components(approximately achromatic and difference of blue-infrared)make up 98% of the signal energy.
Figure 5: Colour parts of the derivative filters used forMSIFT (note that there is an additional near-infrared com-ponent that is not visualised)
Figure 6: RGB-NIR opponent colour components of ascene. The colour transformation converts colour differ-ences into spatial patterns which are well characterised bylocal descriptors such as SIFT. Note that there is visibly lessenergy in the later components.
GIST We compute Gabor filters at 3 scales and 8 orienta-tions per scale. The image is first pre-filtered to nor-malise the local contrast, and the descriptors are con-catenated for each band. We use the implementationfrom Torralba [18].
HMAX Inspired by the “standard model” for recognitionin visual cortex, this consists of a hierarchy of filter-ing and max-pooling operations. HMAX descriptorsare computed independently and concatenated for eachband. We use the CNS implementation by Mutch etal. [15].
The HMAX filters were trained offline using a databaseof 10,000 images. GIST descriptors were normalised perband, which also gave improved performance versus globalnormalisation. In addition to our multispectral SIFT de-
scriptors, we also compare against the best performingcolour descriptor designs proposed by Van de Sande etal. [27, 3], including opponent-SIFT, C-SIFT and rg-SIFT.
7. DatabaseOur test database consists of 477 images distributed
in 9 categories as follows: Country(52), Field(51), For-est(53), Mountain(55), Old Building(51), Street(50), Ur-ban(58), Water(51). The images were processed using auto-matic white balancing for the RGB components, and equalweights on the RGB sensor responses for the NIR compo-nents, with standard gain control and gamma correction ap-plied.
8. Classification ExperimentsTo perform classification based on MSIFT descriptors,
we adopt the method of Boiman et al. [1]. This assumesthat the descriptors zi in image I are i.i.d. Gaussian, withthe class conditional density being approximated by thenearest-neighbour
p(I|c) =�
i
p(zi|c) ⇥�
i
N(zi,NNc(zi), �2) (6)
where NNc(zi) is the nearest neighbour of descriptor zi inclass c of the training set. We then use a Bayes classifier,choosing the class c� = arg maxc p(c|I) with equal classpriors. For the GIST and HMAX descriptors we use LinearSVMs (C-SVC) with a constant c parameter of 100.
9. Experiments (RGB-NIR dataset)We perform scene recognition on our dataset of 477 im-
ages, randomly selecting 99 images for testing (11 per cat-egory) and training using the rest. In all our experimentswe repeat using 10 trials with a randomly selected train-ing/test split, and quote the mean and standard deviation ofthe recognition rate (fraction of correct matches).
Firstly, we experiment with various colour representa-tions for each lifting algorithm. These are: l = luminance(greyscale), li = luminance + NIR, rgb = red, green andblue, rgbi = RGB + NIR, opp = opponent colour space(as used for opponent-SIFT), nopp = normalised opponentcolour space (as used for C-SIFT), lrg = luminance + nor-malised r, g (as used for rg-SIFT), pca1 = 1st RGB-NIRPCA component, pca2 = 1st and 2nd RGB-NIR PCA com-ponents, pca3 = RGB-NIR PCA components 1-3, pca4 =RGB-NIR PCA components 1-4, rnd = random 4�4 lin-ear transform (10 randomised transforms used over 10 tri-als each). The results are shown in Table 1. We name thecombination pca4 sift = MSIFT (the algorithm described insection 5).
![Page 67: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/67.jpg)
Mul0-‐spectral SIFT
• 477 RGB+NIR images -‐ 9 scene categories
• SIFT, GIST, HMAX descriptors. • 11 random images per class for training, rest for tes0ng. -‐ Repeated 10 0mes
•MSIFT (=pca 4) sta0s0cally significantly outperforms the others.
M. Brown and S. Süsstrunk, Mul0-‐spectral SIFT for Scene Category Recogni0on, CVPR 2011.
![Page 68: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/68.jpg)
Mul0-‐spectral SIFT
• 477 RGB+NIR images -‐ 9 scene categories
• SIFT, GIST, HMAX descriptors. • 11 random images per class for training, rest for tes0ng. -‐ Repeated 10 0mes
•MSIFT (=pca 4) sta0s0cally significantly outperforms the others.
M. Brown and S. Süsstrunk, Mul0-‐spectral SIFT for Scene Category Recogni0on, CVPR 2011.
![Page 69: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/69.jpg)
Seman0c Image Segmenta0on
N. Salama0, D. Larlus, G. Csurka, and S. Süsstrunk, Seman0c Image Segmenta0on Using Visible and Near-‐Infrared Channels, in ECCV’s 4th Color and Photometry in Computer Vision Workshop (2012).
Visible NIR V Seg V+NIR Seg GT
![Page 70: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/70.jpg)
Shadow Detec0on
Visible NIR Binary Shadow Mask
D. Rüfenacht, C. Fredembach and S. Süsstrunk, Automatic and accurate shadow detection using near-infrared information, IEEE Transaction on Pattern Analysis and Machine Intelligence (TPAMI), 2014.
![Page 71: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/71.jpg)
Material-‐based Image Segmenta0on
Visible Intrinsic Image
Visible SegmentaBon
VisibleVisible
Visible and NIR SegmentaBon
N. Salama0 and S. Süsstrunk, Material-‐Based Object Segmenta0on Using NIR Informa0on, Proc. IS&T/SID 18th Color Imaging Conference (CIC), 2010.
![Page 72: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/72.jpg)
Conclusions
• In many imaging applica0ons, the spectral dimension is oven ignored. -‐ the assump0on is that the color channels are too correlated.
• That is subop0mal as there are many applica0ons where color (contrast) is key: -‐ Saliency, super-‐pixels,…and many others!
• Beyond the visible spectrum, the “colors” are even less correlated. -‐Near-‐infrared is a prac0cal extension to current capture.
![Page 73: Opponent Colors Revisitedhelper.ipam.ucla.edu/publications/cp2015/cp2015_12523.pdf · CorrelaonbetweenRGBandNIR •10’000RGBNIRpixelpairs,jointentropy: M.BrownandS.Süsstrunk,MulspectralSIFTforSceneCategoryRecognion,"CVPR](https://reader033.fdocuments.us/reader033/viewer/2022042415/5f2fe80e305b535a016021f4/html5/thumbnails/73.jpg)
Acknowledgments
• Saliency: -‐ RK Achanta, S. Hemami, F. Estrada, A. Shaji
• Superpixels/Voxels: -‐ RK Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, G. Yildirim.
• Near-‐Infrared: -‐ M. Brown, C. Fredembach, Z. Sadeghipoor, D. Rüfenacht, N. Salama0, D. Larlus, G. Csurka
• Funding -‐ Swiss Na0onal Science Founda0on, Hasler Founda0on, Xerox Founda0on.
• Publica4ons: ivrg.epfl.ch/publiaBons