Visual Recognition and Searchrogerioferis.com/VisualRecognitionAndSearch2014/classes/class1.pdf ·...

Rogerio Feris, Jan 23, 2014 EECS 6890 – Topics in Information Processing

Spring 2014, Columbia University http://rogerioferis.com/VisualRecognitionAndSearch2014

Visual Recognition and Search

Visual Recognition And Search Columbia University, Spring 2014

Introduction

Instructors:

Rogerio Feris http://rogerioferis.com

Liangliang Cao http://researcher.watson.ibm.com/researcher/view.php?person=

us-liangliang.cao

Jun Wang http://researcher.watson.ibm.com/researcher/view.php?person=

us-wangjun


Introduction

1970s

Early Days of Computer Vision


Introduction

First Digital Camera (1975)

0.01 Megapixels

23 seconds to record a photo to cassette


Introduction

Datasets with 5 or 10 images

Large-Scale Experiment: 800 photos (Takeo Kanade Thesis, 1973)


Introduction

Today

Visual Data is Exploding!


Introduction

Announcement of Pope Benedict in 2005


Introduction

Announcement of Pope Francis in 2013

Rapid proliferation of mobile devices equipped with cameras


Introduction

Billions of cell phones equipped with cameras

~500 billion consumer photos are taken each year world-wide ~500 million photos taken per year in NYC alone

Hundreds of millions of Facebook photo uploads per day

Era of Big Visual Data


Introduction


Introduction

How to annotate large amounts of visual data?

Crowdsourcing: Amazon Mechanical Turk

ImageNet: Millions of Annotated Images, Thousands of Categories

http://www.image-net.org/


Introduction

ImageNet 2012 Challenge

Impressive results obtained by deep convolutional networks


Introduction

http://horatio.cs.nyu.edu/


Introduction

Exciting Time for Computer Vision

+ DATA

+ Computational Processing

+ Advances in Computer Vision and Machine Learning


Introduction

Plan for Today

Visual Recognition and Search: Applications & Challenges

Student Introductions

Course Overview and Requirements

Syllabus Tour

Cool Data for Projects


Applications

Applications

Visual Recognition and Search


Applications

Check Sebastian Thrun short TED talk: http://www.ted.com/talks/sebastian_thrun_google_s_driverless_car.html CVPR 2012 talk with more details: http://techtalks.tv/talks/self-driving-cars/56391/

Self-Driving Cars

Object Recognition and Tracking – mostly 3D point clouds (laser scanner)

Passed Driver’s license test in Nevada!

http://www.ted.com/talks/sebastian_thrun_google_s_driverless_car.html



http://techtalks.tv/talks/self-driving-cars/56391/







Applications

Driver Assistance

Collision Warning Automatic Head Light Control Lane Departure Control etc.

http://www.mobileye.com/

Check interesting talk by Amnon Shashua: http://www.youtube.com/watch?v=cg5SNoTF3Ms


Applications

Seeking the strongest rigid detector (CVPR 2013) Rodrigo Benenson, Markus Mathias, Tinne Tuytelaars, Luc Van Gool


Applications

Mars Exploration Rover

Autonomous Navigation

Highly textured environment – good for stereo!

Terrain Classification

Automatic Photo capture of interesting scenes

Horizontal velocity estimation in landing – feature tracking


Applications

Gaming

Kinect Camera: RGB + Depth sensor

Human pose Detection and Gesture Recognition

See J. Shotton et al, Real-Time Human Pose Recognition in Parts from a Single Depth Image, CVPR 2011 (Best paper award)

Watch for Abnormal Events!


Applications

What did you notice in the video?

People walking?

Vehicles driving?

Abandoned bag???


Applications


Applications

Surveillance

http://www.today.com/video/today/51630165/#51630165

http://www.today.com/video/today/51630165/

12-17-2013 5-43-26 PM.mp4


Applications

People Search in Surveillance Videos


Applications

Video Demo: http://rogerioferis.com/demos/PeopleSearch_BlurredFaces.wmv


Applications

Boston Bombing Event

Suspect #1 found in 4 images in top 8 results Suspect #2 found in 3 images in top page

1071 detected faces from 50 high-res Boston images (all from Flickr)

Ability to spot a person with e.g., a white hat in a crowded scene


Applications

Cashier Fraud Detection: Faked Scans (aka Sweethearting fraud) Giving-away of merchandise without charge to a "sweetheart" customer (e.g., friend, family, fellow employee) by faked product scan Q. Fan et al, Recognition of Repetitive Sequential Human Activity, CVPR 2009

shorter-sweethearting.wmv

Like a normal field guide…

that you can search and sort

and with visual recognition

See N. Kumar et al,

"Leafsnap: A Computer

Vision System for

Automatic Plant Species

Identification, ECCV 2012

Nearly 1 million downloads

40k new users per month

100k active users

1.7 million images taken

100k new images/month

100k users with > 5 images

Users from all over the world

Botanists, educators, kids, hobbyists,

photographers, …

Slide Credit: Neeraj Kumar


Applications

Google Goggles

Amazon


Applications

Auto-focus by Face Detection http://www.rogerioferis.com/demos/FaceTrackerRealTime.avi

TAAZ Virtual Makeover http://www.taaz.com/


Applications

IntraFace Demo (http://www.humansensing.cs.cmu.edu/intraface/)


Applications

Finding Visually Similar Objects


Applications

Exploring Community Photo Collections http://phototour.cs.washington.edu/


Applications

Organizing Photo Collections – IBM IMARS http://researcher.watson.ibm.com/researcher/view_project.php?id=877


Applications

Many More Applications…

Medical Imaging

Biometrics

Handwritten Digit Recognition

And so on ….


Challenges

Does it really work?

Many problems still remain to be solved…


Challenges

Slide Credit: Kristen Grauman


Break


Student Introductions

Introductions

Student Info Form (see website: Homework #0) Send by Monday, Jan 27


Course Overview

Course Goals:

Understand and analyze the state-of-the-art in visual recognition and search.

Hands-on experience with cutting-edge

methods for visual recognition and search.

Note: 7:00pm – 8:50pm (3 credits)


Course Overview

Pre-requisites:

1) Courses/Background in Computer Vision or Image Processing is mandatory. 2) Programming skills in C/C++ or Matlab are also required. Please drop the class if you don’t satisfy these requirements


Course Overview

Course Format:

Each class will consist of:

Lecture by instructor Student presentations and discussion

Paper Reviews Solo (no groups)

Project

Final project report: Paper (4-8 pages)


Course Overview

Grading:

Class Participation (10%)

Paper Presentations (20%)

Paper Reviews (20%)

Term Project (50%)


Course Overview

Resources:

Check course webpage for:

Publicly Available Source Code Datasets Related Courses


Course Overview

Presentations:

Students will be divided in groups and will have to prepare one or two presentations in class (list of presentation topics will be provided soon).

More information about the format and length/time of the presentations will be available in the course webpage.


Course Overview

Presentations: Presentation slides will have to be sent to instructors at least

one day before the presentation.

Grading based on clarity, organization, knowledge, and whether the presentation meets the time constraints.

Check http://www.slideshare.net/cameraculture/how-to-give-a-good-talk

http://www.slideshare.net/cameraculture/how-to-give-a-good-talk













Course Overview

Paper Reviews: Individual evaluation (no groups)

Each review (1-2 pages) should address paper strengths,

weaknesses, quality of experiments, and describe ways in which the paper could be extended.

Check the course webpage for more details


Course Overview

Projects:

Groups of two or three depending on the total number of students attending the class.

Students are expected to spend significant time on their projects

Project Proposal, Project Update I, Project Update II, Final Presentation, and Final Project Report (4-8 pages)


Course Overview

Projects:

Projects will be graded according to the following criteria:


Syllabus Tour

Syllabus Tour

Broad set of topics; focus on the state-of-the-art

Check the schedule and required reading for each class in the course webpage


Syllabus Tour

Part I: From Low-level to Semantic Visual Representations


Syllabus Tour

Low-Level Representation: Feature Detection and Description

Modern Descriptors for Real-Time Applications: FAST, BRISK, …

Check ECCV 2012 Tutorial on Modern Features: https://sites.google.com/site/eccv12features/slides

Classical Descriptors: SIFT and SURF


Syllabus Tour

Mid-level feature coding and pooling

Classical Bag-of-Word Models Spatial Pyramid Matching

Modern higher-level representations:

Fisher vector and super-vector encoding, LLC, … [K. Chatfield et al, The devil is in the details: an evaluation of recent feature encoding methods, BMVC 2011]


Syllabus Tour

Encoding Structure: Part-based Models

Deformable Part Models (Felzenszwalb et al)

Structureless Rigid

Poselets (Bourdev et al)

Check ICCV 2013 Tutorial on Part-based models for Recognition: http://www.cs.berkeley.edu/~rbg/ICCV2013/


Syllabus Tour

High-level Representation: Attributes and Semantic Features

Output of Semantic Classifiers as Features, Zero-shot Learning, …

Check CVPR 2013 Tutorial on Attributes: https://filebox.ece.vt.edu/~parikh/attributes/


Syllabus Tour

Part II: Tools for Large-scale Image Classification and Retrieval


Syllabus Tour

Large-Scale Classification

Insights on how to handle huge amounts of data and categories

Deep Convolutional Neural Networks

Check NIPS 2013 Tutorial on Deep Learning: http://cs.nyu.edu/~fergus/presentations/nips2013_final.pdf


Syllabus Tour

Fast Indexing and Image Retrieval

Metric Learning

Indexing via Learning-based Hashing

Database


Syllabus Tour

Active Learning


Syllabus Tour

Part III: Case Studies


Syllabus Tour

Case Studies

IBM Smart Vision Suite (SVS) IBM Multimedia Analysis and Retrieval System (IMARS)


Syllabus Tour

NOT Covered:

Basic Machine learning algorithms. We assume you are familiar with e.g., SVMs, …

Computer vision fundamentals. We assume you are familiar with e.g., edge detection, …


Citizen Science Projects




http://www.youtube.com/watch?v=AUL03ivS8bY http://www.snapshotserengeti.org/

Snapshot Serengeti

http://www.youtube.com/watch?v=AUL03ivS8bY

http://www.youtube.com/watch?v=AUL03ivS8bY

http://www.snapshotserengeti.org/

Serengeti1-21-2014 1-22-15 PM.mp4



5 Terabytes of annotated data



Example Project: Build a web demo for fine-grained categorization of animals, using

state-of-the-art visual learning techniques

Top classification results and associated confidence: 1. Zebra (0.98) 2. Striped Hyena (0.01) 3. Leopard (0.01)

We expect a comprehensive experimental analysis for each project


Past 2013 Projects

Check the previous year project presentations


How to Invent:

See http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010

http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010





Next Steps

Don’t forget to submit Homework #0 (Student Info Form) if you are taking this class

Background in computer vision + coding skills in

Matlab or C/C++ are strict requirements for taking the course

If you don’t satisfy these requirements, please drop the class as soon as possible, as it will help us to know the final number of students in class.

Thank you!

Visual Recognition and Searchrogerioferis.com/VisualRecognitionAndSearch2014/classes/class1.pdf ·...

Documents

Transcript of Visual Recognition and Searchrogerioferis.com/VisualRecognitionAndSearch2014/classes/class1.pdf ·...