ICPSR Biennial Meeting October 2, 2015 Ryan Womack ...€¦ · to present a 3-D surface...

(A bit about) Data VisualizationICPSR Biennial Meeting

October 2, 2015

Ryan Womack ([email protected])Data Librarian, Rutgers University

This work is licensed under a Creative Commons Attribution

-NonCommercial-ShareAlike 4.0 International License.

Ryan Womack ([email protected]) Data Librarian, Rutgers University(A bit about) Data Visualization 1 / 52

http://creativecommons.org/licenses/by-nc-sa/4.0/

http://creativecommons.org/licenses/by-nc-sa/4.0/

Introduction

What this talk IS:

Discusses standard techniques of data visualization, the day-to-daypower tools for understanding data

Reviews various graphical techniques, from early to recent, fromsimple to advanced

Presents principles of good data presentation, and show the Rimplementation of many functions


Introduction

What this talk is NOT:

It is not about “infographics”, the beautiful, heavily customizedproducts of expert graphic designers. [See 1 and 2 for morediscussion]

It is not about the cognitive science aspects of data perception[wish I knew more about this!]

It is not about how to use R or other software [although code isprovided for those who are interested]

It is not necessarily a balanced survey of all data visualization. Inparticular, it is light on graph networks, clustering, and trees [notmy expertise]

Very little mapping, too [Others are better at this]


http://www.stat.columbia.edu/~gelman/research/published/vis14.pdf

http://www.stat.columbia.edu/~gelman/research/published/visreply3.pdf

Setup

Most of the graphics examples that are not web accessible are runin R.

R is open source software available at http://r-project.org

RStudio is a useful freely available editor available athttp://rstudio.com

Workshop materials, including R scripts, supplemental images anddata, are available for download fromhttp://ryanwomack.com/ICPSR2015

The R script file contains working demonstrations of many of theconcepts mentioned here for you to try on your own.


http://r-project.org

http://rstudio.com

http://ryanwomack.com/ICPSR2015

Outline

Why?

Whirlwhind tour of historical data viz

Standard visualization vs. some less commonly used examples

3-D and Animation

Interactivity, data exploration

A little bit of big data


Why Data Visualization?

Data visualization can:

provide clear understanding of patterns in data

detect hidden structures in data

condense information


Anscombe’s Quartet

For example, see Anscombe’s quartet (image source:http://commons.wikimedia.org/wiki/File:Anscombe%27s quartet 3.svg):


Links to DataViz sites

Some examples of good data visualization (and fancy infographics) canbe found at:

Information Aesthetics

Chart Porn

Eagereyes

DataVis.ca

VizWiz

US Census Data Visualization Gallery


http://infosthetics.com

http://chartporn.org

http://eagereyes.org

http://www.datavis.ca/

http://vizwiz.blogspot.ca/

http://www.census.gov/dataviz/

Bad Graphs

Pie Charts are known to be problematic

Clutter and other issues can ruin graphics

Novel or nonsensical?

For more bad ideas, try:

Junk Charts

Ten Worst Graphs

WTFviz


http://sharpstatistics.co.uk/charts/multiple-pie-charts/

http://www.stevefenton.co.uk/Content/Pie-Charts-Are-Bad/

http://junkcharts.typepad.com/junk_charts/2013/03/which-software-is-responsible-for-this.html

http://andrewgelman.com/2015/08/30/another-bad-chart-for-you-to-criticize/

http://junkcharts.typepad.com

http://www.biostat.wisc.edu/~kbroman/topten_worstgraphs/

http://viz.wtf

Pie Chart Examples

image source: http://peltiertech.com/WordPress/3d-pie-charts/


Pie Chart Examples

image source: http://ndevisual.wordpress.com/tag/uses-of-pie-charts/


Pie Chart Examples

image source: http://www.nbcchicago.com/news/local/FOX-News-Chart-Fails-Math-73711092.html


Pie Chart Examples

image source: http://tips.vovici.com/content/111031 swb


Clutter Example

image source:http://junkcharts.typepad.com/junk charts/2013/03/which-software-is-responsible-for-this.html


Playfair

Astronomical observations, charts, and maps led in graphicalinnovation prior to 1800. See also Classic Data Visualizations

William Playfair is the pioneer of the line chart, bar chart, timeseries plots, and pie chart.

Playfair, W. (1786). Commercial and Political Atlas: Representing, byCopper-Plate Charts, the Progress of the Commerce, Revenues, Expenditure,and Debts of England, during the Whole of the Eighteenth Century,

Playfair, W. (1801). Statistical Breviary.

Both republished in The Commercial and Political Atlas and StatisticalBreviary, 2005, Cambridge University Press.


http://davegiles.blogspot.com/2015/08/classic-data-visualizations.html

http://blog.plot.ly/post/117105992082/time-series-graphs-eleven-stunning-ways-you-can

http://blog.plot.ly/post/117105992082/time-series-graphs-eleven-stunning-ways-you-can

Playfair Examples


Minard

Charles Joseph Minard was the next influential data graphic creatorafter Playfair.

Minard’s flow map of Napoleon’s Russian campaign is celebratedby Tufte and others as one of the greatest information graphics.

It embodies an ideal of highly compressed informative elements,presented with style

Six variables: size, location in 2 dimensions, the direction of thearmy, temperature, date [and group]

However, this is a one-off design that crosses into Infographics, butit can be reproduced in R and other software.


http://vannevar.blogspot.com/2009/03/minard-napolean-russia-1812-best-chart.html

http://www.datavis.ca/gallery/re-minard.php

Minard Examples


Fisher and Tukey

In the 20th century, statisticians such as Ronald Fisher and JohnTukey continued to advance graphical methods for the analysis ofdata.

Fisher emphasized plotting the data to understand relationships.

Tukey’s Exploratory Data Analysis emphasized the use of graphicsto understand the data during analysis, rather than the finalpresentation to an outside audience.

Tukey created the box and whiskers plot and the stem and leafplot.


Tufte

Edward R. Tufte’s series of books, beginning with The Visual Displayof Quantitative Information, have become the most widely know workson data visualization.

There is considerable overlap between the various publications

Tufte’s ideal is highly compressed, elegant, and informative data,as expressed in dense printed graphics

Tufte sometimes emphasizes beauty and design to the detriment ofsimplicity and clarity [e.g., train schedules]

“Graphical elegance is often found in simplicity of design andcomplexity of data.”

“Beautiful graphics do not traffic with the trivial.”


Train Schedule from Marey


Tufte’s principles

Tufte has developed and popularized numerous principles andterminology:

Graphics reveal data - show the data without distorting it - “above allelse show the data”

Small multiple - understanding one slice makes understanding otherseasier

Lie factor - effect shown/effect in reality

Graphical Integrity - no lies, let data vary, not design

Data density - maximize data/ink ratio

Sparklines - seems they haven’t caught on

chartjunk - self-explanatory

Powerpoint is responsible for most of the world’s sorrows [TheCognitive Style of Powerpoint]


Lie Factor

image source: http://www.datavis.ca/gallery/lie-factor.php


Cleveland

William Cleveland’s Elements of Graphing Data and VisualizingData pioneered systematic considerations of data legibility

Cleveland is particularly known for promoting the dot plot as aalternative to bars and pies.

The dot plot provides clarity and easy comparison of data.

Cleveland also pioneered Trellis graphics

Trellis graphics emphasizes comparison of multiple panels of data

The lattice package implements Trellis graphics in R

See Cleveland.pdf for a summary of Cleveland’s recommendations


http://www.stat.purdue.edu/~wsc/

http://ryanwomack.com/IASSIST/DataViz/Data/Cleveland.pdf

Scatterplot matrix


The Grammar of Graphics

The Grammar of Graphics, by Leland Wilkinson, was extremelyinfluential in thinking about graphics

Grammar means ”rules for art and science”

The Grammar of Graphics specifies rules both mathematical andaesthetic

Earlier graph producers focused on aesthetics of static content

Dynamic graphics and scientific visualization, by contrast, requiresophisticated designs to enable brushing, drill-down, zooming,linking

The Grammar of Graphics is easily adapted to this approach

ggplot2 was developed by Hadley Wickham as an implementationof the Grammar of Graphics


From Barchart to Dot Plot

The Cleveland dot plot

use to compare labeled quantities, ordered lists


Figure: Bar chart v. Dot Plot


Visualizing Distributions of Data

Box and Whiskers Plot

illustrate quantiles and outliers. There is also a Tufte version.

Violin plot

Blends density information with box and whiskers style (in anartistic manner)


http://stackoverflow.com/questions/6973394/functions-available-for-tufte-boxplots-in-r

Figure: Box Plot v. Violin Plot


Visualizing Categorical Data

Beyond the pie chart

The mosaic plot allows multiple categories to be displayed on thesame graph, but can be complicated to interpret.

The spineplot is a variant of the mosaic plot, plotting proportionsin 2 dimensions.


Figure: Pie Chart v. Mosaic Plot


Maps and Glyphs

Maps are obviously an important and widespread way of presentingdata.

We examine a few examples of choropleth maps, in which shadingindicates data levels

See also Interactive Maps in R and 5 kinds of Interactive maps inPlot.ly for further exploration

Glyphs present iconic representations of data elements.

Weather maps often use glyphs.

A more dynamic example is here.

As an R example, consider Chernoff faces and the aplpack

package. Also, Smiley faces [and many more graph variants in thischapter].


http://r-video-tutorial.blogspot.com/2015/05/interactive-maps-for-web-in-r.html

http://blog.plot.ly/post/128662310992/analyze-data-five-ways-you-can-make-interactive

http://blog.plot.ly/post/128662310992/analyze-data-five-ways-you-can-make-interactive

http://weather.gc.ca/canada_e.html

http://hint.fm/wind/

http://statacumen.com/teach/SC1/SC1_18_Plots.pdf

Figure: Choropleth Map v. Chernoff Faces


3-D

3-D scatterplots

cloud (lattice)

contour plots

to plot standardized levels of data

wireframe plots

to present a 3-D surface representation of data

rgl (a separate package containing several 3d plotting functionsand animation)

mosaic3d extends the mosaic paradigm to three dimensions


Animation

Animation is an easy way to step through data over time

or to provide comparisons of different views of data

R makes animation easy with the animation package

Just enclose a sequence of graphics in the animation command togenerate interactive HTML (or GIF, SWF, LATEX, Video).


Interactive DataViz - Principles

Why aren’t all of our graphs interactive?

Brushing is used to select data points and track them throughvarious analyses.

Drilling down, zooming, and subsetting are also interactivetechniques.

Data displays can be linked so that a selection in one panelmodifies the output displayed in another panel.

Interactivity is especially useful for data exploration, studyingmultidimensional relationships.


http://www.statsblogs.com/2013/03/16/why-arent-all-of-our-graphs-interactive/

Interactive Data in Practice

There are many R packages that allow for interactive data work in agraphical user interface, including:

playwith - versatile package that works with any graphicsfunction. Graphics can be explored, edited, and exported.

requires separate installation of GTK+ on your computer [method variesby OS]


googleVis

In many contexts, visualizing the relationships between data elementsis made easier by viewing related data interactively.

Making this easy are googleVis and other “Vis” packages, e.g.bdvis for biodiversity or rainfreq.

A Library example - comparing selected ARL Statistics for publicCIC universities


https://vijaybarve.wordpress.com/2014/05/08/package-bdvis-is-on-cran/

http://rationshop.blogspot.com/2014/05/new-r-package-rainfreq-rainfall.html

http://ryanwomack.com/ICPSR2015/CIC.html

http://arlstatistics.org

Interactive Data on the Web - Rcharts

Rcharts is a package that uses javascript to create interactivevisualizations.

Lattice-style commands are used.

The package can output javascript for use in an HTML page.

Some commands depend on supplemental javascript libraries thatmust be installed, such as NVD3

Can embed in documents too, with slidify


http://rcharts.io/

http://nvd3.org/

Interactive Data on the Web - shiny

The shiny package is developed by the Rstudio folks

You can learn shiny in half a day via the online tutorial

More custom control of the design is possible with shiny, incomparison to other do-it-all packages

Graphics use familiar R syntax (including ggplot2), with wrappersto implement web functionality

Every shiny app has the same structure: two R scripts savedtogether in a directory [ui and server files]

You must install the shiny server to deliver pages via the web


http://shiny.rstudio.com

http://blog.rstudio.org/2015/06/16/shiny-0-12-interactive-plots-with-ggplot2/

Interactive Data on the Web - shiny, cont.

There are samples built into the shiny package.

You can build a Census Explorer of your own with theseinstructions from Ari Lamstein.

You can see more in the shiny gallery

Rcharts works with shiny too.


https://arilamstein.shinyapps.io/census-explorer

http://www.arilamstein.com/blog/2015/08/06/tutorial-recap-make-a-census-explorer-with-shiny/

http://www.arilamstein.com/blog/2015/08/06/tutorial-recap-make-a-census-explorer-with-shiny/

http://shiny.rstudio.com/gallery/

Interactive Data on the Web - ggvis

The ggvis package is ALSO developed by the Rstudio folks

Think ggplot meets shiny

Similar syntax to ggplot

Some ability to add interactive controls

Can embed in shiny for web access


http://ggvis.rstudio.com

Interactive Data on the Web - radiant

Radiant is another new R interface built with shiny

The following links demonstrate capabilities:

vnijs.shinyapps.io/basevnijs.shinyapps.io/quantvnijs.shinyapps.io/marketing

By automating the mechanics of interacting with data, we canfocus on exploring and understanding.


http://vnijs.github.io/blog/2015/05/introducing-radiant.html

http://vnijs.shinyapps.io/base

http://vnijs.shinyapps.io/quant

http://vnijs.shinyapps.io/marketing

Other (non-R) options for Web visualization

D3.js, free at http://d3js.org/

Inkscape, free at https://inkscape.org/

Tableau, free 1-year student license athttp://www.tableau.com/academic/students

Plot.ly environment at http://plot.ly


http://d3js.org/

https://inkscape.org/

http://www.tableau.com/academic/students

http://plot.ly

Interactive Power

Population pyramids are one example whereinteractivity + animation = insight .

Populationpyramid.net - for all countries, basic animation

The German Population Pyramid from Destatis is even moreinteractive

Doing it in R is possible with these instructions (Part 1) and (Part2)

The ggvis package is ALSO developed by the Rstudio folks


http://populationpyramid.net/

https://www.destatis.de/bevoelkerungspyramide/

http://walkerke.github.io/2014/06/rcharts-pyramids/

https://rstudio-pubs-static.s3.amazonaws.com/71818_7748b38431934f60944c287563e0c7df.html

https://rstudio-pubs-static.s3.amazonaws.com/71818_7748b38431934f60944c287563e0c7df.html

http://ggvis.rstudio.com

Big Data

Big data presents special issues for data visualization

While many techniques and graphics are the same, explorationand plotting must be optimized for the size of the data set

Representation of the complexity of the data may require specialtechniques

hexbin

bigvis


bigvis

bigvis was an experimental package by Hadley Wickham to deal withthe issues of Big Data

There is a Preprint and R Meetup presentation by Hadley Wickham

Complete code is available at https://github.com/hadley/bigvis-infovis

Target: process 100 million observations in under 5 seconds.

Fundamental principle: No need for more data points than there arepixels on the screen.

“ggstat” package has been mentioned as a future project that willincorporate these ideas.


http://vita.had.co.nz/papers/bigvis.html

http://www.livestream.com/knerd/video?clipId=pla_45d98657-44b7-4227-86c3-01332879fa54&utm_source=lslibrary&utm_medium=ui-thumb

https://github.com/hadley/bigvis-infovis

bigvis steps

Condense (bin, condense)

Smooth (smooth, best_h, peel)

Visualize (autoplot plus standard methods)


Trelliscope (Tessera)

Tessera is developed by Purdue, Pacific Northwest NationalLaboratory, and Mozilla. Launched in November 2014, this projectholds a lot of promise.

Running in the R environment, Tessera provides its own commands thatexecute across a cluster, easing the burden of analysis in this environment.

The datadr package “divides and recombines” in a manner similarto MapReduce, providing a simplified interface to Hadoop.

Tessera has its own visualization interface, Trelliscope, that canhandle views across many variables and observations. Described inthis paper.

Tessera’s Bootcamp is a good introduction, or try the quickstart.

Live demo is here.


http://tessera.io

http://tessera.io/docs-datadr/#introduction

http://tessera.io/docs-trelliscope/

http://ml.stat.purdue.edu/docs/trelliscope.ldav.2013.pdf

http://tessera.io/docs-r-intro-bootcamp/

http://tessera.io/#quickstart

https://hafen.shinyapps.io/tesseraTutorial/#name=list_sold_vs_time_quickstart&group=common&labels=county,state

Infographics links

Although not covered here, the following links are a sampling ofinfographics sites for your later enjoyment:

Data Storytelling in Video

Art of Data Visualization - in spite of its title, more on theinfographics side

Parisian Subway Traffic and New York Subway Inequality

Tulp Interactive

Mapping London and London Riots + Twitter

YouTube Trends Map

Global Burden of Disease Visualizations

and the Tree of Life


http://eagereyes.org/blog/2013/data-storytelling-video

http://spatial.ly/2013/05/art-data-visualisation-video/

http://metropolitain.io/

http://www.newyorker.com/sandbox/business/subway.html

http://tulpinteractive.com/

http://life.mappinglondon.co.uk/

http://www.guardian.co.uk/uk/interactive/2011/dec/07/london-riots-twitter

http://www.youtube.com/trendsmap

http://www.healthdata.org/gbd/data-visualizations

http://www.opentreeoflife.org/

Keep Exploring

Data Visualization represents a nearly infinite world of possibilty forexploration:

plunge into programming

deep dives into data

indulge in interactivity

...have fun and keep learning! [e.g., R-bloggers.com]


http://www.r-bloggers.com

References

There is also an online bibliography of references to accompany thispresentation on my home page.


http://ryanwomack.com/IASSIST/DataViz/Data/DataVizBib.pdf

ICPSR Biennial Meeting October 2, 2015 Ryan Womack ...€¦ · to present a 3-D surface...

Documents

Transcript of ICPSR Biennial Meeting October 2, 2015 Ryan Womack ...€¦ · to present a 3-D surface...