Video and Image Analytics for Business, Consumer and ... · Video and Image Analytics for Business,...
Transcript of Video and Image Analytics for Business, Consumer and ... · Video and Image Analytics for Business,...
Video and Image Analytics for
Business, Consumer and Social
Insights
Jialie Shen
School of Information Systems, SMU
Outline
• Part I: General introduction
– What is image and video mining (analytics)?
– Why they are so important to business?
• Part II: Novel business applications
– Online advertisement
• Rich media advertising
– Retail customer analytics
• Visual tracking for monitoring customers in-house behaviours
• Part III: Conclusion & QA
– Proactive thoughts, open issues and future directions
2
General Introduction
• Modern ITs have led to fast growth of large video &
image collections
• Analyzing image & video to discover useful knowledge is
becoming important
– manual processing is important due to huge volume
• Goal of image and video analytics (IVA)
– developing intelligent computer program to automatically gain
• comprehensive understandings of data content
• effective/scalable data management
3
General Introduction (cont.)
• IVA research is a hot area ……
• Technologies developed are generally used to
perform two kinds of tasks
– Descriptive mining – find out the general properties of
the data in the database (e.g., image clustering)
– Predictive Mining – perform inference on the current
data in order to make predictions (e.g., age and emotion
estimation based on face)
4
General Introduction (cont.)
• IVM research is a hot area ……
• Technologies developed are generally used to
perform two kinds of tasks
– Descriptive mining – find out the general properties of
the data in the database (e.g., image clustering), and
– Predictive Mining – perform inference on the current
data in order to make predictions (e.g., age and emotion
estimation based on face)
5
Descriptive Mining – Image Clustering
6
Assuming we have a
set of images without
any structure
Descriptive Mining – Image Clustering
7
Raw image set
Clustering
Algorithm
After passing through clustering algorithm, ........
Descriptive Mining – Image Clustering
8
Raw image set
Waterfall cluster
Sun cluster
Clustering
Algorithm
After passing through clustering algorithm, the
images are grouped into two clusters
General Introduction (cont.)
• IVA research is a hot area ……
• Technologies developed are generally used to
perform two kinds of tasks
– Descriptive mining – find out the general properties of
the data in the database (e.g., image clustering), and
– Predictive Mining – perform inference on the current
data in order to make predictions (e.g., age and emotion
estimation based on face image)
9
Predictive Mining - Age Estimation
10
Face Images
Assuming we have a set of face images as inputs
to age estimation system ........
Predictive Mining - Age Estimation
11
Age estimation system
extracts visual features
from images
Face Images
Predictive Mining - Age Estimation
12
Machine
Learning
Algorithm
AGE
(25, 18, ?)
Supervised or Semi-Supervised
Predictor
Feature Extraction
(Gabor Filtering)
Face Images
Using visual
features, ML
algorithm
predicts age
General Introduction (cont.)
• IVM research is a hot area ……
• Technologies developed are generally used to
perform two kinds of tasks
– Descriptive mining – find out the general properties of the
data in the database (e.g., scene analysis), and
– Predictive Mining – perform inference on the current data
in order to make predictions (e.g., age estimation based on
visual appearance)
13
Why images and videos
are important to business?
General Introduction (Cont.)
– “One picture worth a thousand words”
• a complex business idea can be conveyed with just a single still
image or video sequence
• Image and video can bring your customer more information
14
15
What is this image???
Message: ????
Who is this person????
16
Uncle Sam (initials U.S.) is a
common national personification
of the American government
Who:
Message: Join us! Buddy! We need u!....
What?
Very classic advertisement used
for promoting US army
recruitment in WWI and WW II.
General Introduction (Cont.)
– “One picture worth a thousand words”
• a complex business idea can be conveyed with just a single
still image and video sequence
• Image and video can bring your customer more information
– Image and video are very helpful for people to retain
business information
• People can remember 10% of the information they read
20% of the information they hear, but
50% of the information they see.
17
18
Perhaps no one knows accurate number!
General Introduction (Cont.)
19
6 billion (Aug 2011)
• 192 years to view all of them (1s per image)
• 3000+ uploads/minute
• 2% Internet users visit (2009)
• Daily time on site: 4.7 minutes (2009)
690 million (Mar 2012)
• 3,450 years to see all of them
• 48 hours uploaded/minute (2012)
• 20% Internet users visit (2009)
• Daily time on site: 23 minutes (2009)
100 billion (Middle of 2011 )
• 3,200 years to view all of them (1s per image)
• ~200M uploads/day; ~ 6B/month (2012)
• 800+M users (Dec 2011)
• Daily time on site: 30 minutes (2009)
General Introduction (Cont.)
20
We have millions online images and
videos
They could be wonderful carriers for
business messages
if we know how to use them effectively
???
Outline
• Part I: General introduction
– What is image and video mining (analytics)?
– Why they are so important to business people?
• Part II: Novel business applications
– Online advertisement
• Rich media advertising
– Retail customer analytics
• Visual tracking for monitoring customer in-house behaviours
• Part III: Conclusion & QA
– Proactive thoughts, open issues and future directions
22
Online Advertising
• A sub-area of advertising/marketing
• Power the revenue of a large number of Internet
service companies
• Provide financial support for free content/service
• Many online ads formats
– Text ads
– Rich media ads
– …
23
Online Advertising
• A sub-area of advertising/marketing
• Power the revenue of a large number of Internet
service companies
• Provide financial support for free content/service
• Many online ads formats
– Text ads
– Rich media ads
– …
24
Online Rich Media Ads
25
• No formal definition yet!
• A range of interactive digital media including video, audio and
associated text for online marketing campaigns.
• An emerging online monetization strategy to achieve various goals
Importance of Rich Media Ads
• Reshaping for online marketing
• Evolution of Internet has brought fundamental changes in
the way people obtain and generate biz info.
– Online media contents surged to an unprecedented level
due to the Web 2.0
• Rich media becomes the primary force to drive marketing
campaign
• Richmedia ads possess many unique advantages
– Better capability to convey business information
– More attractive and salient than plain text ads
– Prefect for driving purchase intent
26
An Exemplary System
27
• A contextual in-image advertizing platform
– Aims to embed “right” visual-based ads into online images
• To achieve the goal, the system can
– match ads to image content and related text information
automatically
• “contextual relevance” – which image ads to be selected?
– find “right” insertion positions
• “optimal position” - where to insert ads?
An Exemplary System (cont.)
28
• Three major components
Relevance
Optimization
Multimodal
Information
Extraction &
Analysis
Vision based
Web Page
Analysis
An Exemplary System (cont.)
29
Vision-based Web
Page Analysis
Web Pages
Image: Position:100,200
Size: 500x250 URL:http://www.nytimes.
com/imagepages/
2010/01/23/sports/
Crawler
Surrouding Text: Kobe, NBA, LA Lakers,
New York, Basketball,
Sports, ……
• Vision-based web page analysis
has two functionalities.
• Given a web page with images, it • selects the images suitable for
advertising, and
• extracts surrounding text
An Exemplary System (cont.)
30
Vision-based Web
Page Analysis
Multimodal Information
Extraction & Analysis
Web Pages
Image: Position:100,200
Size: 500x250 URL:http://www.nytimes.
com/imagepages/
2010/01/23/sports/
Crawler
Surrouding Text: Kobe, NBA, LA Lakers,
New York, Basketball,
Sports, ……
Local + Gobal Textual Information
High-level
Concept Detector
Insertion Point
Detection
Text Data
Filtering
Visual Information Analysis
High level
concept
detection using
surrounding text
Image analysis for
determining ads
insertion point
• Two sub-modules • Text analysis module
• Image analysis module
An Exemplary System (cont.)
31
Vision-based Web
Page Analysis
Multimodal Information
Extraction & Analysis Relevance
Optimization
Text
Relevance
Match
Ad Rank List Ads
Ads
Reranking
Module
Web Pages
Image: Position:100,200
Size: 500x250 URL:http://www.nytimes.
com/imagepages/
2010/01/23/sports/
Crawler
Surrouding Text: Kobe, NBA, LA Lakers,
New York, Basketball,
Sports, ……
Local + Gobal Textual Information
High-level
Concept Detector
Insertion Point
Detection
Text Data
Filtering
Visual Information Analysis
Relevance optimization module uses the high level
concept gained via surrounding text analysis to get
“right” ad via ranking ads based on relevance.
An Exemplary System (cont.)
32
Vision-based Web
Page Analysis Multimodal Information
Extraction & Analysis
Relevance
Optimization
Text
Relevance
Match
Ad Rank List Ads
Ads
Reranking
Module
Web Pages
Image: Position:100,200
Size: 500x250 URL:http://www.nytimes.
com/imagepages/
2010/01/23/sports/
Crawler
Surrouding Text: Kobe, NBA, LA Lakers,
New York, Basketball,
Sports, ……
Local + Gobal Textual Information
High-level
Concept Detector
Insertion Point
Detection
Text Data
Filtering
Visual Information Analysis
The optimal insertion point determined by
visual information analysis module tells us
where is the ‘”right” place to insert “right” ad
Experience Gained
• Effective solution = media analysis + text mining +
business domain knowledge
• A hugh unexplored domain:
– How to achieve the best trade-off advertizing strategy making
different users happy at the same time?
– How to maximize its business or economic impacts under
different metrics?
What is best business model (e.g. Cost per Impression)
• Industrial partner’s involvement is IMPORTANT!
33
Outline
• Part I: General introduction
– What is image and video mining (analytics)?
– Why they are so important to business people?
• Part II: Novel business applications
– Online Advertisement
• Rich media advertising
– Retail customer analytics
• Visual tracking for monitoring customers in-house behaviours
• Part III: Conclusion & QA
– Proactive thoughts, open issues and future directions
34
Visual Tracking (VT) – Customer Traffic
35
• The goal of every retailer: sales profit maximization
• Understanding customers is essential.
• Tracking customers can provide a lot off
information about customer in-store movement
patterns & appearance
Visual Tracking – Customer Traffic
• Labour based security is expensive
• Cameras could be very helpful to reduce cost
– But, watching the video feeds is tedious
• Labour based security is ineffective
– dedicated and well motivated personnel can't stay focused on
monitoring tasks for more than 20 minutes
• VT techniques are very useful to increase the business
performance via analyzing customer s’ in-store
behaviors automatically
36
Visual Tracking – Customer Traffic
37
• With VT technology, we can automatically
– Count/aggregate the number of people passing
through certain spots
• valuable to improve customer service by adapting
business resources to match customer traffic
– Track the number of customers during different days
and hours
• allowing optimal displaying configuration for in-store
promotions and events
Visual Tracking – Customer Behavior
38
• With VA technique, we also can
– Analyze store traffic to identify the major customer
trajectories
• determine general strategies for product placement
– Identify the regions that attract less customers
• optimize store layout to best satisfy customers and drive sales
Outline
• Part I: General introduction
– What is image and video mining (analytics)?
– Why they are so important to business people?
• Part II: Novel business applications
– Online advertisement
• Rich media advertising
– Retail customer analytics
• Visual tracking for monitoring customer in-house behaviours
• Part III: Conclusion & QA
– Proactive thoughts, open issues and future directions
39
Conclusion and Q&A
• We have reviewed
– what is image and video analysis
– why they are so important to us
– two business applications
• Analysing image and video is one of the most
appealing research directions for business
intelligence
40
Conclusion and Q&A
• Why this is an interesting domain for scholars in
management science/CS or CEO?
• Before answering the question, we need to have a
look another set of questions
– Who should be real god for a firm?
– Over long run, where do firm’s cash flows come from? (how should we measure business value of a firm)
• For sure, from your customers
• Where /how do you estimate cash flow from customer & predict
customer’s life time value?
41
Acknowledgement
42
• The research study is kindly supported by
Thank you very much for your kind
attentions and valuable time!