1
Visual Processing for Social Media
Andrew C. GallagherTsuhan Chen
September 30, 2012
Cornell University
Outline
Social Media Overview Visual Processing Overview Social Media Insights Within the Image Social Media Insights From Sharing
2
3
Now, pictures of people Examples of how social data has helped
understand images of people Some things I’ve learned about people
from computer vision
4
Understanding images of people
5
What the computer sees
6
Faces in the lab
[Turk et al., Cog. Neuro. 1991]
[Belhaumer et al., PAMI 1997]
[Wiskott et al., PAMI, 1997] [Lucey et al., IJCV 2007]
[Blanz et al., PAMI 2003]
[Kanade, Kyoto U. 1973]
8
Is this a family?
9
The Loop
Images and Computer Vision
What we know about people
10
Understanding Images of People
Describe people: How tall? How old? Identify people: Who? Why are they together? Exploit the same context humans use!
11
Understanding Images of People
Capture Context Social Context
July 2, 20058:27 PMLat: 42.2902Long: 85.5361
June 25, 200510:50 AMLat: 42.3202Long: 85.1261
12
Understanding Images of People
Capture Context Social Context
Adult male height: 177 cmAdult female height: 163 cmMLE mother-child: 27 yearsMLE husband-wife: 2 yearsMLE |sibling-sibling|: 6 years
13
What is social context?Social Context: information about people and their society that is useful for understanding images. Distributions of ages and genders in social
groups Social relationships Face position in a group image First name popularity over time Anthropometric measurements
14
Group Images What we know and learn about people:
Group dynamics Computer vision task:
Measuring age, gender, of each person in a group
15
Images of Groups
Identify age and gender Recognize certain group events Consider context and appearance
[A. Gallagher, T. Chen, CVPR 2009]
16
Contextual Features
Absolute face position Size, position relative neighbor and group Minimal spanning tree degree
17
Evidence of Social Context
Relative positions of nearest neighbors depends on the social relationship
Mean distance is 306 mm
Neighbors Male to Female Other to Baby
19
Evidence of Social Context
Samples of faces based on image locationRandom samples
20
Use All Context
5080 images with 28,231 faces Classification improves with more
contextual features
21
Appearance Features
Project face into Fisher Space Nearest neighbor density estimation
Gender subspaceNearest neighbors
22
Gender Estimation
Context Appearance Combined
24
Context and Appearance
Context contributes more when appearance is weak.
Context Appearance Combined
All Faces Small Faces
25
Context for Scene Geometry
Find the face vanishing lineEstimated horizon from face positionsManually labeled horizon
26
Context for Dining Event
Group Structure = Activity
27
Row Segmentation
[A. Gallagher, T. Chen, ICME 2009]
29
Row Segmentation
30
Row Segmentation
31
Social Relationship Retrieval
Spouses
Mother-Child
[G. Wang, A. Gallagher, J. Luo, D. Forsyth, ECCV 2010]
32
Names as Context What we know and learn about people:
Government census data Computer vision task:
Matching names to faces. Guessing age and gender.
33
First names capture information about age and gender
First names are social context
Person A and Person B
First Names as Context
Mildred and Lisa
1900 1920 1940 1960 1980 20000
0.01
0.02
0.03
0.04
0.05
0.06
0.07
Birth Year
Pro
babi
lity
Probability of Birth Year
LisaMildred
1900 1920 1940 1960 1980 20000
0.01
0.02
0.03
0.04
0.05
0.06
0.07
Birth Year
Pro
babi
lity
Probability of Birth Year
MildredLisaNoraPeytonLinda
Source: U.S. Social Security Administration
[A. Gallagher, T. Chen, CVPR 2008]
34
First names and appearance? Tom_101 Ben_165 Caleb_337 Andrew_233 Brian_116 Zachary_431
1953 1956 2003 1984 1962 1996
Abigail_194 Heather_224 Alejandra_152 Juanita_192 Ethel_165 Gertrude_532002 1970 1977 1947 1926 1924
35
Gertrude_531924
Sort by Expected Age Tom_101
1953Ben_165
1956
Caleb_3372003
Andrew_2331984
Brian_1161962
Zachary_4311996
Abigail_1942002
Heather_2241970
Alejandra_1521977
Juanita_1921947
Ethel_1651926
36
First Names as Context
Mildred and Lisa
Name
Birth Year
AgeFeatures
Gender
GenderFeatures
Image-Based Features
37
More Context = Better Results
Appearance First Name Full Model
38
Recognition from First Name
The model improves name assignment, age estimation, and gender classification
39
Learning about people
39
Images and Computer Vision
What we know about people
40
Group Images How close do people stand in group
photos? Computer vision answer: 306 mm Sociology’s “Personal Space”: 457 mm
Do people suspend personal space needs during photograph?
41
Group Images: Gender Prior How do people end up in a group photo
anyhow?
42
Group Images: Gender Prior Bernoulli world?
Implicit prior, IID: Let’s look at the data!
43
“Group Shots”
Number of Females
Gender Distribution of 6 peopleBinomial Distribution
Number of Females 0 1 2 3 4 5 6
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0 1 2 3 4 5 60
0.05
0.1
0.15
0.2
0.25
0.3
0.35
?
0 1 2 3 4 5 60
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Number of Females
?
“Family”
Actu
al D
istrib
ution
s
Genders of people in a image are not independent!
44
Group Shot Analysis Standing Order Frequency for 4 people (2
male, 2 female):0.13
0.11
0.19
0.13
0.30
0.15
But why?
45
Learning about people
4545
Images and Computer Vision
What we know about people
(what they do and think!)
46
Social Context Data Summary U.S. Social Security First Name Database
6693 first names, birth years, gender U.S. CDC National Center for Health
Statistics Physical growth tables Birth rates and other birth statistics Family structure statistics
Farkas, 1994 Facial anthropometric measurements
47
Conclusions Social context is useful for interpreting
single images or image collections Social context is learned from images or
other public sources Learning about people improves our
understanding of images of people
48
Related Publications J. Xin, A. Gallagher, L. Cao, J. Luo, J. Han
The Wisdom of Social Multimedia: Using Flickr For Prediction and Forecast
ACM MM 2009
G. Wang, A. Gallagher, J. Luo, D. Forsyth
Seeing People in Social Context: Recognizing People and Social Relationship
ECCV 2010
A. Gallagher, A. Blose, T. Chen
Jointly Estimating Demographics and Height with a Calibrated Camera
ICCV 2009
A. Gallagher, T. Chen Using Context to Recognize People in Consumer Images IPSJ Trans. on Comp. Vis. and Apps., 2009
A. Gallagher, T. Chen Understanding Images of Groups of People CVPR 2009
A. Gallagher, T. Chen Finding Rows of People in Group Images ICME 2009
A. Gallagher, C. Neustaedter, J. Luo, L. Cao, T. Chen
Image Annotation Using Personal Calendars as Context ACM MM 2008
A. Gallagher, T. Chen Estimating Age, Gender and Identity using First Name Priors
CVPR 2008
A. Gallagher, T. Chen Clothing Cosegmentation for Recognizing People CVPR 2008
P. Singla, H. Kautz, J. Luo, A. Gallagher
Discovery of Social Relationships in Consumer Photo Collections Using Markov Logic
CVPR SLAM 2008
A. Gallagher, T. Chen Using a Markov Network to Recognize People in Consumer Images
ICIP 2007
A. Gallagher, M. Das, A. Loui
User-Assisted People Search in Consumer Image Collections ICME 2007
A. Gallagher, T. Chen Using Group Prior to Identify People in Consumer Images CVPR SLAM 2007
Top Related