Visible PartisanshipPolmeth XXXIII, Rice University, July 22, 2016Convolutional Neural Networks for the Analysis of Political Images

L. Jason Anastasopoulos ljanastas@uga.edu (University of Georgia, Public Admin + Policy, Political Science, Georgia Informatics Institute)Dhruvil Badani (UC Berkeley, EECS)Crystal Lee (UC Berkeley, EECS)Shiry Ginosar (UC Berkeley, EECS)

Outline

▪ Background

▪ Image experiment - “the people you pose with”– how race and gender of people politicians pose with affect perceptions.

▪ Race classifier for images using convolutional neural networks.

▪ Analysis of race in US House of Representative Facebook profile photos.

Visual semantics: image elements

▪ Symbols

▪ Objects

▪ People

▪ Poses

Images convey political meaning: symbols

Images convey political meaning: objects

Source - usa4palin.com: Still from Sarah Palin’s Amazing America

Images convey political meaning: people

Source - haaretz.com: Netanyahu (left), Obama (middle), Abbas (right)

Images convey political meaning: poses

Source - google.com: Image search for “John Boehner”

Political functions of images

For politicians…▪ Signaling

▪ Partisanship/ideology.▪ Policy positions.

▪ Homestyle (Fenno 1978)▪ Qualification – competence.▪ Identification – “I am one of you.”▪ Empathy – “I care about your needs.”

Political functions of images

For politicians…▪ Signaling

▪ Partisanship/ideology.▪ Policy positions.

▪ Homestyle (Fenno 1978)▪ Qualification – competence.▪ Identification – “I am one of you.”▪ Empathy – “I care about your needs.”

Political functions of imagesFor news media…

▪ Issue framing.▪ Persuasion.

Hardware and software limitations

▪ Hardware▪ Even small images are “big data.”

▪ One 200 x 200 image = 3 200x200 matrices or 1 vector of length 120,000.

Hardware and software limitations

▪ Software▪ High dimensional statistical theory developed more recently.

▪ Asymptotics deals with properties of estimators as with a fixed number of parameters, p.

▪ In modern machine learning applications,

Introductionclassical asymptotic theory: sample size n → +∞ with number ofparameters p fixed

modern applications in science and engineering:! large-scale problems: both p and n may be large (possibly p ≫ n)! need for high-dimensional theory that allows (n, p) → +∞

Introductionclassical asymptotic theory: sample size n → +∞ with number ofparameters p fixed

modern applications in science and engineering:! large-scale problems: both p and n may be large (possibly p ≫ n)! need for high-dimensional theory that allows (n, p) → +∞

Image analysis renaissance in social science

Hardware: Powerful CPUs and now GPUs in desktop computers (thanks gamers!)

Image analysis renaissance in social science

▪ Software▪ Statistical theory for computing in high

dimensions.

▪ Advances in numerical computing.

▪ Deep-learning frameworks: Torch, Tensorflow, Theano, Caffe.

Signaling and image features

▪ Symbols

▪ Objects

▪ People

▪ Poses

Signaling and image features

▪ Symbols

▪ Objects

▪ People

▪ Poses

Questions

▪ How do the group characteristics (gender, race, age, etc.) of people that Members of Congress pose with affect how they are perceived?

▪ Social media “homestyle”▪ Is there evidence that Members of Congress use social media images to signal

identification and empathy with constituents using group characteristics?

Questions

▪ How do the group characteristics (gender, race, age, etc.) of people that Members of Congress pose with affect how they are perceived?

▪ Social media “homestyle”▪ Is there evidence that Members of Congress use social media images to signal

identification and empathy with constituents using group characteristics?

“The people you pose with” experiment: Lou Barletta

▪ Lou Barletta (R-PA, 11) chosen for initial experiment because of relative obscurity and similar pictures with different groups of people.

▪ MTurk respondents randomly assigned one of 7 images with Barletta▪ Alone – Barletta by himself.

▪ Woman – Barletta with a woman.

▪ Man – Barletta with a white man.

▪ Black – Barletta with African-American men.

▪ Asked a series of questions based only on the image.

Image treatmentsAlone Man Woman Af. American

What is your best guess of the political party that this politician belongs to?Beliefs about Barletta’s party ID vary significantly by image shown.

Alone: 39% guessed Democrat61% guessed Republican

Black: 58% guessed Democrat42% guessed Republican

Man: 42% guessed Democrat58% guessed Republican

Woman: 43% guessed Democrat57% guessed Republican

What is your best guess of this politician’s ideological orientation?Average by treatment groups

Alone: Moderate.

Black: Liberal.

Man: Moderate.

Woman: Moderate.

Does this politician seem…honest and trustworthy?Perceived to be more trustworthy when pictured next to a woman.

Does the politician seem…like a strong and decisive leader?Perceived to be a stronger leader when pictured next to a woman.

Does the politician seem…knowledgeable about the issues?Perceived to be less knowledgeable when pictured next to an older white man.

Does the politician seem…like someone who shares my values? (non-white respondents)Perceived by non-white respondents to share their values when pictured next to African-American men.

Barletta experiment conclusions

▪ Opinion of Barletta affected by group identity of individuals included in images.

▪ Race affected beliefs about partisanship/ideology and implied “shared values.”

▪ Gender affected beliefs in trustworthiness, honesty and decisiveness.

▪ Survey experiment expanding – goal is to test which aspects of photos most strongly tied to perceptions of candidate ideology and party.

Questions

▪ How do the group characteristics (gender, race, age, etc.) of people that Members of Congress pose with affect how they are perceived?

▪ Social media “homestyle”▪ Is there evidence that Members of Congress use social media images to signal

identification and empathy with constituents using group characteristics?

Questions

▪ How do the group characteristics (gender, race, age, etc.) of people that Members of Congress pose with affect how they are perceived?

▪ Social media “homestyle”▪ Is there evidence that Members of Congress use social media images to signal

identification and empathy with constituents using group characteristics?

Data

300,000+ Facebook images with text posts for accounts of:

300 US House members.

56 US Senate members.

Goals and Methods

▪ Identify race of individuals pictured in Facebook profiles of House Members.▪ Viola-Jones Algorithm▪ Train a convolutional neural network race classifier.

▪ Explore how distribution of racial groups in photos compare to congressional district demographics, partisanship and ideology.▪ Compare Facebook profile “demographics” with district demographics, party id

and DW-Nominate scores.

Results

▪ Democrats and Republicans in the US House of Representatives have very different social media styles.

▪ Evidence that Democrats use Facebook images to elicit racial identification and empathy among constituents.

How you see an image…▪ Image as data.

▪ Eg 620x412 pixel image.

You see:{Donald Trump, blue tie, black suit, blue background, anger}

How a computer sees an image...▪ 640x412 pixel

image.

▪ 3 Channels: Red, Green, Blue

▪ 3 640x412matrices of pixel intensity values.

▪ -or- 3x640x412 = 791,040 x 1 vector

Human image classification is robust

Machine image classification is error prone...

Image Credit: Andrej Karpathy

Theoretical means of image feature extraction limited mostly to faces...

Viola-Jones Object Detection Framework

Data driven approach/supervised machine learning approach – train models utilizing pixel intensity data...

Collect labeled images.

Train a machine learning classifier.

Test classifier accuracy.

CIFAR-10 library of 32x32 labels images benchmark performance.

One layer neural network

X1

X2

X3

▪ Inputs multiplied by weights and added create “hidden” layer.

▪ Hidden layer passed through “activation function” multiplied by another set of weights to generate class probabilities/scores.

▪ Simplest model discussed by psychologist Rosenblatt (1958)

Neural network activation functions

▪ Most common activation functions are sigmoid and tangent.

▪ Optimizing predictions requires▪ Choice of activation

functions and;

▪ Choice of weights.

Backpropagation

▪ Rumelhart, Hinton and Williams (1986)

▪ Selection of weights involves:▪ Forward pass - Calculation of loss function.

▪ Backward pass – Use of chain rule and stochastic gradient descent to iteratively calculate new weights.

Convolutional neural networks for image feature classification

Multi-layer neural network involving series of activation functions on chunks of pixel data.

Convolutional neural networks for image feature classification• Equivalent of

passing pixel data through a number of “filters.”

• Discover which “filter” is activated by which labeled image category.

• Output is highest probability category given filter responses

Image credit: Andrej Karpathy

Convolutional neural network: building a race classifier

• Labeled image data

• 60,000 high school yearbook images, 1960-2013• 6,000 images sampled from Congressional Facebook dataset we collected.• Categories: White, African-American, East Asian, Hispanic.

• 16-layer CNN model for large-scale image recognition from CNN Model Zoo by Simonyan and Zisserman (2015): http://arxiv.org/pdf/1409.1556.pdf and https://gist.github.com/ksimonyan/211839e770f7b538e2d8#file-readme-md

Convolutional neural network: building a race classifier

• Step 1: Identify faces using Viola-Jones algorithm.

• Image on the right is a Facebook photo from Representative Tammy Duckworth ‘s (D-IL) profile.

Convolutional neural network: building a race classifier

• Step 2: Label race of faces.

Convolutional neural network: building a race classifier

• Step 3: Train CNN on labeled data.

Convolutional neural network: building a race classifier

• Step 4: Test classifier accuracy

• Avg. cross-validated accuracy rates of 90% for whites, 85% for African-American, 75% for Asian, 65% for HIspanic.

Convolutional neural network: building a race classifier

• Step 5: Estimate race of individuals in Congressional Facebook image set using trained model.

Race and partisanship in House Facebook image posts (white House members)

White Democrats post Facebook photos of …

African-Americans at 4x the rate of white Republicans

Hispanics at 1.2x the rate of white Republicans.

Asians at 2x the rate of white Republicans.

Race and partisanship in House Facebook image posts (white House members)Even conditional on relevant district demographics, state and regionfixed effects, evidence of conscious efforts by partisans to include/exclude racial groups in Facebook image posts.

Democrats (white): +6% more African-Americans in posts.

Republicans (white): +6% more whites in posts.

Identification and empathy: district demographics and Facebook image posts

Do MCs strategically post photos of racial groups to engender identification and empathy from constituents?

Overall strong evidence that they do.

Strong relationship between % of racial group in district and % of racial group posted in Facebook profiles.

Identification and empathy: district demographics and Facebook image posts by party

Strategic use of race in image posts much more evident among Democrats than Republicans

Identification and empathy: district demographics and Facebook image posts by party

Y = % white in Facebook profile photos

White Democrats more “race conscious” whenposting FB photos.

After conditioning on state and region fixed effects and district demographics, Democrats Facebook photos more likely to reflectracial/ethnic mix of district.

Identification and empathy: district demographics and Facebook image posts by party

Representation = % White in Facebook profile photos –% White in Congressional District

Whites over-represented in Facebook photosof white Democrats and Republicans…

Identification and empathy: district demographics and Facebook image posts by party

Representation = % Black in Facebook profile photos –% Black in Congressional District

African-Americans under-representedin Facebook photos of Republican MCsby an average of about 3.8%

Identification and empathy: district demographics and Facebook image posts by party

Representation = % Hispanic in Facebook profile photos% Hispanic in Congressional District

Hispanics under-representedIn Facebook photos of both parties, more so among Democrats

Identification and empathy: district demographics and Facebook image posts by party

Representation = % Asian in Facebook profile photos% Asian in Congressional District

Asians under-represented in Facebookphotos of white Democrats.

Discussion

• Modern computational methods allow for the large scale analysis of images.

• Here we build a race classifier for images using convolutional neural networks.

Discussion

▪ Characteristics of people that politicians pose with shape perceptions.

▪ Democrats and Republicans in the US House of Representatives have very different social media styles.

▪ Evidence that Democrats use Facebook images to elicit racial identification and empathy among constituents.