Visual Recognition and Searchrogerioferis.com/VisualRecognitionAndSearch2014/classes/class1.pdf ·...

Rogerio Feris, Jan 23, 2014 EECS 6890 – Topics in Information Processing

Spring 2014, Columbia University http://rogerioferis.com/VisualRecognitionAndSearch2014

Visual Recognition and Search

Visual Recognition And Search Columbia University, Spring 2014

Introduction

Instructors:

Rogerio Feris http://rogerioferis.com

Liangliang Cao http://researcher.watson.ibm.com/researcher/view.php?person=

us-liangliang.cao

Jun Wang http://researcher.watson.ibm.com/researcher/view.php?person=

us-wangjun

Introduction

Early Days of Computer Vision

Introduction

First Digital Camera (1975)

0.01 Megapixels

23 seconds to record a photo to cassette

Introduction

Datasets with 5 or 10 images

Large-Scale Experiment: 800 photos (Takeo Kanade Thesis, 1973)

Introduction

Visual Data is Exploding!

Introduction

Announcement of Pope Benedict in 2005

Introduction

Announcement of Pope Francis in 2013

Rapid proliferation of mobile devices equipped with cameras

Introduction

Billions of cell phones equipped with cameras

~500 billion consumer photos are taken each year world-wide ~500 million photos taken per year in NYC alone

Hundreds of millions of Facebook photo uploads per day

Era of Big Visual Data

Introduction

How to annotate large amounts of visual data?

Crowdsourcing: Amazon Mechanical Turk

ImageNet: Millions of Annotated Images, Thousands of Categories

http://www.image-net.org/

Introduction

ImageNet 2012 Challenge

Impressive results obtained by deep convolutional networks

Introduction

http://horatio.cs.nyu.edu/

Introduction

Exciting Time for Computer Vision

+ DATA

+ Computational Processing

+ Advances in Computer Vision and Machine Learning

Introduction

Plan for Today

Visual Recognition and Search: Applications & Challenges

Student Introductions

Course Overview and Requirements

Syllabus Tour

Cool Data for Projects

Applications

Visual Recognition and Search

Applications

Check Sebastian Thrun short TED talk: http://www.ted.com/talks/sebastian_thrun_google_s_driverless_car.html CVPR 2012 talk with more details: http://techtalks.tv/talks/self-driving-cars/56391/

Self-Driving Cars

Object Recognition and Tracking – mostly 3D point clouds (laser scanner)

Passed Driver’s license test in Nevada!

Applications

Driver Assistance

Collision Warning Automatic Head Light Control Lane Departure Control etc.

http://www.mobileye.com/

Check interesting talk by Amnon Shashua: http://www.youtube.com/watch?v=cg5SNoTF3Ms

Applications

Seeking the strongest rigid detector (CVPR 2013) Rodrigo Benenson, Markus Mathias, Tinne Tuytelaars, Luc Van Gool

Applications

Mars Exploration Rover

Autonomous Navigation

Highly textured environment – good for stereo!

Terrain Classification

Automatic Photo capture of interesting scenes

Horizontal velocity estimation in landing – feature tracking

Applications

Gaming

Kinect Camera: RGB + Depth sensor

Human pose Detection and Gesture Recognition

See J. Shotton et al, Real-Time Human Pose Recognition in Parts from a Single Depth Image, CVPR 2011 (Best paper award)

Watch for Abnormal Events!

Applications

What did you notice in the video?

People walking?

Vehicles driving?

Abandoned bag???

Applications

Surveillance

http://www.today.com/video/today/51630165/#51630165

Applications

People Search in Surveillance Videos

Applications

Video Demo: http://rogerioferis.com/demos/PeopleSearch_BlurredFaces.wmv

Applications

Boston Bombing Event

Suspect #1 found in 4 images in top 8 results Suspect #2 found in 3 images in top page

1071 detected faces from 50 high-res Boston images (all from Flickr)

Ability to spot a person with e.g., a white hat in a crowded scene

Applications

Cashier Fraud Detection: Faked Scans (aka Sweethearting fraud) Giving-away of merchandise without charge to a "sweetheart" customer (e.g., friend, family, fellow employee) by faked product scan Q. Fan et al, Recognition of Repetitive Sequential Human Activity, CVPR 2009

Like a normal field guide…

that you can search and sort

and with visual recognition

See N. Kumar et al,

"Leafsnap: A Computer

Vision System for

Automatic Plant Species

Identification, ECCV 2012

Nearly 1 million downloads

40k new users per month

100k active users

1.7 million images taken

100k new images/month

100k users with > 5 images

Users from all over the world

Botanists, educators, kids, hobbyists,

photographers, …

Slide Credit: Neeraj Kumar

Applications

Google Goggles

Amazon

Applications

Auto-focus by Face Detection http://www.rogerioferis.com/demos/FaceTrackerRealTime.avi

TAAZ Virtual Makeover http://www.taaz.com/

Applications

IntraFace Demo (http://www.humansensing.cs.cmu.edu/intraface/)

Applications

Finding Visually Similar Objects

Applications

Exploring Community Photo Collections http://phototour.cs.washington.edu/

Applications

Organizing Photo Collections – IBM IMARS http://researcher.watson.ibm.com/researcher/view_project.php?id=877

Applications

Many More Applications…

Medical Imaging

Biometrics

Handwritten Digit Recognition

And so on ….

Challenges

Does it really work?

Many problems still remain to be solved…

Challenges

Slide Credit: Kristen Grauman

Student Introductions

Introductions

Student Info Form (see website: Homework #0) Send by Monday, Jan 27

Course Overview

Course Goals:

Understand and analyze the state-of-the-art in visual recognition and search.

Hands-on experience with cutting-edge

methods for visual recognition and search.

Note: 7:00pm – 8:50pm (3 credits)

Course Overview

Pre-requisites:

1) Courses/Background in Computer Vision or Image Processing is mandatory. 2) Programming skills in C/C++ or Matlab are also required. Please drop the class if you don’t satisfy these requirements

Course Overview

Course Format:

Each class will consist of:

Lecture by instructor Student presentations and discussion

Paper Reviews Solo (no groups)

Project

Final project report: Paper (4-8 pages)

Course Overview

Grading:

Class Participation (10%)

Paper Presentations (20%)

Paper Reviews (20%)

Term Project (50%)

Course Overview

Resources:

Check course webpage for:

Publicly Available Source Code Datasets Related Courses

Course Overview

Presentations:

Students will be divided in groups and will have to prepare one or two presentations in class (list of presentation topics will be provided soon).

More information about the format and length/time of the presentations will be available in the course webpage.

Course Overview

Presentations: Presentation slides will have to be sent to instructors at least

one day before the presentation.

Grading based on clarity, organization, knowledge, and whether the presentation meets the time constraints.

Check http://www.slideshare.net/cameraculture/how-to-give-a-good-talk

Course Overview

Paper Reviews: Individual evaluation (no groups)

Each review (1-2 pages) should address paper strengths,

weaknesses, quality of experiments, and describe ways in which the paper could be extended.

Check the course webpage for more details

Course Overview

Projects:

Groups of two or three depending on the total number of students attending the class.

Students are expected to spend significant time on their projects

Project Proposal, Project Update I, Project Update II, Final Presentation, and Final Project Report (4-8 pages)

Course Overview

Projects:

Projects will be graded according to the following criteria:

Syllabus Tour

Broad set of topics; focus on the state-of-the-art

Check the schedule and required reading for each class in the course webpage

Syllabus Tour

Part I: From Low-level to Semantic Visual Representations

Syllabus Tour

Low-Level Representation: Feature Detection and Description

Modern Descriptors for Real-Time Applications: FAST, BRISK, …

Check ECCV 2012 Tutorial on Modern Features: https://sites.google.com/site/eccv12features/slides

Classical Descriptors: SIFT and SURF

Syllabus Tour

Mid-level feature coding and pooling

Classical Bag-of-Word Models Spatial Pyramid Matching

Modern higher-level representations:

Fisher vector and super-vector encoding, LLC, … [K. Chatfield et al, The devil is in the details: an evaluation of recent feature encoding methods, BMVC 2011]

Syllabus Tour

Encoding Structure: Part-based Models

Deformable Part Models (Felzenszwalb et al)

Structureless Rigid

Poselets (Bourdev et al)

Check ICCV 2013 Tutorial on Part-based models for Recognition: http://www.cs.berkeley.edu/~rbg/ICCV2013/

Syllabus Tour

High-level Representation: Attributes and Semantic Features

Output of Semantic Classifiers as Features, Zero-shot Learning, …

Check CVPR 2013 Tutorial on Attributes: https://filebox.ece.vt.edu/~parikh/attributes/

Syllabus Tour

Part II: Tools for Large-scale Image Classification and Retrieval

Syllabus Tour

Large-Scale Classification

Insights on how to handle huge amounts of data and categories

Deep Convolutional Neural Networks

Check NIPS 2013 Tutorial on Deep Learning: http://cs.nyu.edu/~fergus/presentations/nips2013_final.pdf

Syllabus Tour

Fast Indexing and Image Retrieval

Metric Learning

Indexing via Learning-based Hashing

Database

Syllabus Tour

Active Learning

Syllabus Tour

Part III: Case Studies

Syllabus Tour

Case Studies

IBM Smart Vision Suite (SVS) IBM Multimedia Analysis and Retrieval System (IMARS)

Syllabus Tour

NOT Covered:

Basic Machine learning algorithms. We assume you are familiar with e.g., SVMs, …

Computer vision fundamentals. We assume you are familiar with e.g., edge detection, …

Citizen Science Projects

http://www.youtube.com/watch?v=AUL03ivS8bY http://www.snapshotserengeti.org/

Snapshot Serengeti

5 Terabytes of annotated data

Example Project: Build a web demo for fine-grained categorization of animals, using

state-of-the-art visual learning techniques

Top classification results and associated confidence: 1. Zebra (0.98) 2. Striped Hyena (0.01) 3. Leopard (0.01)

We expect a comprehensive experimental analysis for each project

Past 2013 Projects

Check the previous year project presentations

How to Invent:

See http://www.slideshare.net/cameraculture/raskar-ideahexagonapr2010

Next Steps

Don’t forget to submit Homework #0 (Student Info Form) if you are taking this class

Background in computer vision + coding skills in

Matlab or C/C++ are strict requirements for taking the course

If you don’t satisfy these requirements, please drop the class as soon as possible, as it will help us to know the final number of students in class.

Thank you!

Visual Recognition and Searchrogerioferis.com/VisualRecognitionAndSearch2014/classes/class1.pdf ·...

Documents

Transcript of Visual Recognition and Searchrogerioferis.com/VisualRecognitionAndSearch2014/classes/class1.pdf ·...

1. casting class1

Gtrainfreqonly-class1.rvd - RadViz. gtrainfreqonly-class1(1).rvd - RadViz.

JAVA CLASS1

Class1 FINAL

Class1 - Syllabus Course

Class1 dgbl ineducationjhu_summer2013

Class1 Ppt

Class1 Intro+Fund

Class1 110812

Diversity week5 class1

525- class1

Class1 Intro

Class1 Mod

Class1 Introduction

Sdc9 class1 mar11

WV Class1-entry1Part2

WCM - basics - Class1

4 Attention Class1

Virology Class1

IIPersonalProperty Finding Class1