Introduction to Big Data - Harvard University€¦ · Introduction to Big Data Chapter 10 (Week 6)...

Introductionto Big Data

Chapter 10 (Week 6)Exploratory Data Analysis (Visualization)

DCCS208(02) Korea University 2019 Fall

Asst. Prof. Minseok Seomins@korea.ac.kr

Contents

Summary Statistic

Exploratory Data Analysis1. Visualization

Diverse plots

Single Variable Visualization2.

Diverse plots

Visualization for Two Variables 3.

Diverse plots

Visualization for More than Two Variables 4.

01Exploratory Data AnalysisSummary statistic & Visualization

EDA and VisualizationExploratory Data Analysis

Exploratory Data Analysis (EDA) and Visualization are very important steps in any analysis task.

Get to know your data! distributions (symmetric, normal, skewed) data quality problems outliers correlations and inter-relationships subsets of interest suggest functional relationships

Sometimes EDA or visualization could be the goal!

Exploratory Data AnalysisDefinition of EDA

Goal: Get a general sense of the data means, medians, quantiles, histograms, boxplots You should always look at every variable - you will learn

something!

Think interactive and visual Humans are the best pattern recognizers You can use more than 2 dimensions!

x,y,z, space, color, time….

Especially useful in early stages of data mining Detect outliers (e.g. assess data quality) Test assumptions (e.g. normal distributions or skewed?) Identify useful raw data & transforms (e.g. log(x))

Bottom line: it is always well worth looking at your data!

Exploratory Data AnalysisSummary Statistic

Summary statistic is not visualization Sample statistics of data X

Mean: �̅�𝑥= ∑i Xi / n Mode: most common value in X Median: X=sort(X), median = Xn/2 (half below, half above) Quartiles of sorted X: Q1 value = X0.25n , Q3 value = X0.75 n

Interquartile range: value(Q3) - value(Q1)Range: max(X) - min(X) = Xn - X1

Variance: σ2 = ∑i (Xi - �̅�𝑥)2 / n Skewness: ∑i (Xi - �̅�𝑥)3 / [ (∑i (Xi - �̅�𝑥)2)3/2 ]

Zero if symmetric; right-skewed more common Number of distinct values for a variable

Exploratory Data AnalysisInformation Visualization

Information visualization: concerned with data that does not have a well-defined representation in 2D or 3D space (i.e., “abstract data”).

Visualization: converting raw data to a form that is viewable and understandable to humans.

Exploratory Data AnalysisInformation Visualization

Information visualization: concerned with data that does not have a well-defined representation in 2D or 3D space (i.e., “abstract data”).

Visual Encoding VariablesImportant components in visualization

Position Length Area Volume Value Texture Color Shape Transparency Blur / Focus...

Information in Hue and ValueImportant components in visualization

Value is perceived as ordered

Encode ordinal variables (O)

Encode continuous variables (Q)

Encoce nominal variables (N) using colorHue is normally perceived as unordered

Bertin’s Levels of OrganizationImportant components in visualization

Effectiveness RankingImportant components in visualization

By using the key elements of this visualization, you canaccelerate the transformation of information into knowledge.

02Single Variable VisualizationSingle!!!

HistogramSingle Variable Visualization

Shows center, variability, skewness, modality, outliers, or strange patterns.

Bin width and position matter

Beware of real zeros

Pictures of Data: Continuous VariablesHow to make a Histogram

Consider the following data collected from the 1995 StatisticalAbstracts of the United States

• For each of the 50 United States, the proportion ofindividuals over 65 years of age has been recorded

Let’s find out Max and Min values

Break the data range into mutually exclusive, equally sized “bins”:This example used 1% wide.

Let’s count the number of observations in each bin

Pictures of Data: Continuous VariablesDrawing the histogram based on these information

Pictures of Data: HistogramsAnother example

Suppose we have a sample of blood pressure data on a sampleof 113 men

Sample mean (�̅�𝑥) : 123.6 mmHg

Sample Median (Med): 123.0 mmHg

Sample sd (s): 12.9 mmHg

Pictures of Data: Continuous VariablesDrawing the histogram based on these information

Pictures of Data: Continuous VariablesDifferent bin?

Importance of IntervalsBin size

How many intervals (bins) should you have in a histogram?

• There is no perfect answer to this

• Depends on sample size n

• Rough rule of thumb: # Intervals ≈ 𝑛𝑛

Issues with HistogramSingle Variable Visualization

For small data sets, histograms can be misleading. Small changes in the data, bins, or anchor can deceive

For large data sets, histograms can be quite effective atillustrating general properties of the distribution.

Histograms effectively only work with 1 variable at a time.

BoxplotsSingle Variable Visualization

Shows a lot of information about a variable in one plot Median IQR Outliers Range Skewness

Limitations Overplotting It is hard to tell distributional

shape No standard implementation

in software (many options for whiskers, outliers)

SampleMedian

75th Percentile

25th Percentile

LargestObs.

SmallestObs.

Example) Hospital length of stay dataBoxplot

LargeOutliers

Text cloudSingle Categorical Variable Visualization

Sequence LogoSingle Categorical Variable Visualization

Network plot between wordsSingle Categorical Variable Visualization

03Visualization for Two VariablesTwo variables

ScatterplotsFor two continuous variables

Standard tool to display relation between two continuousvariables.

Useful to answer Are X and Y related each other?

Linear Quadratic Other

Variance of Y variable depend on X? Outliers present?

Is there a relationship between X and Y variables?

Variation in Y differs depending on the value of X.

Limitation

It is very difficult to represent a lot of data at once.

Contour plotsFor two continuous variables (Large scale data)

Contour plots are great for representing relationships betweentwo continuous variables.

It doesn’t give you the exact location of each value, but you cansee the relationship and density between two variables.

Two techniques in visualizationFor large scale data

Transparent plotting

Jittering

Histogram with different colorOne continuous and cateogircal variables

If one variable is categorical, we can use small multiples.

‘Color’ and ‘Shape’ can visually represent different category!

Side-by-side boxplotOne continuous and cateogircal variables

Box-plot can likewise represent a single categorical variables asindependent boxes.

Barcharts and SpineplotsOne continuous and cateogircal variables

Stacked barcharts can be used to compare continuous valuesacross two or more categorical ones.

Pie chartsOne continuous and cateogircal variables

Very popular visualization way.

This is good for showing simple relations of proportions.

Barplots, histograms usually better (but less pretty)

04VisualizationMore than two variables

Pairwise scatterplotsMore than two variables

Pairwise scatterplots can represent multiple variables at once.

However, there can be difficulties in expressing categoricalvariables.

Multivariate visualizationMore than two variables

Creative thinking will be required to visualize multiple variables atthe same time.

Conditioning on variables

Trellis or lattice plots Different colors and shapes Infinite possibilities

Simple questionMore than two variables

How many dimensions are represented here?

Parallel Coordinate plotMore than two variables

Networks plotMore than two variables

Visualizaing networks is helpful, even if is not obvious that anetwork exists.

HeatmapMore than two variables

Heatmaps are one of the widely used visualization methods.

InteractivityImportant visualization aporaches these days

As the world-wide web is becoming more common these days,the importance of interaction is growing in the fild of visualization.

Multi-dimentional plotMore than two variables

One variable represents one dimension.

To represent three variables at once, three-dimensional space isrequird.

Multi-dimentional plotMore than two variables

Multi-dimentional plot seems to be fancy.

But there is a practical issue due to the ‘Viewpoint’.

We end up visualizing in 2d spaces like apper or web page.

https://plot.ly/r/3d-scatter-plots/

Dimension reductionVisualization

What if you need to visualize more than 1000 variables?

One possible way is to visualize high dimensional data is toreduce it to 2 or 3 dimensions.

Variable selection (Feature selection)

Principle Components (PCA analysis)

Multi-dimensional scaling

This is next topic!

End of Slide

Introduction to Big Data - Harvard University€¦ · Introduction to Big Data Chapter 10 (Week 6)...

Documents

Transcript of Introduction to Big Data - Harvard University€¦ · Introduction to Big Data Chapter 10 (Week 6)...

The BIG Future of BIG Data · 2017-03-08 · The BIG Future of BIG Data. Big Data Data Governance Data Warehousing Data Reporting Data Infrastructure. Become a PREDICTIVEEnterprise

Big Data Technology Big Data - aakritsubedi9.com.npaakritsubedi9.com.np/files/Big Data Technology.pdf · Big Data Technology Big Data 1"Big data" is a field that treats ways to analyze,

Big Data Meets Big Data Analytics 105777

E6893 Big Data Analytics Lecture 11: Linked Big Data ...cylin/course/bigdata/EECS6893-BigData... · E6893 Big Data Analytics – Lecture 11: Linked Big Data: ... E6893 Big Data Analytics

· for executive: box big data ussuiu lla:ansnnns1ðxnu big data -big data big -wifñuiaÖ big data big data • hadoop big clouderâ manager hive impala big data 22 airuntju 2559

BIG DATA, SMART DATA AND BIG ANALYSIS

An Introduction of Big data; Big data for beginners; Overview of Big Data; Big data Tutorial

Real Time Big data Applications: file · Web viewUNIT I. INTRODUCTION TO BIG DATA. Big Data – Definition, Characteristic Features – Big Data Applications - Big Data vs. Traditional

Big Data Meets Big Data Analytics

Introduction to Big Data. Reference: What is “Big Data”?What is “Big Data”?

Introduction to Big Data, Big Data Processing, and Big ...cis.csuohio.edu/~sschung/CIS660/Lecture1_IntroBigDataAnalyrics.pdf · What’s Big Data? From Wikipedia: • Big data is

Big Success With Big Data - Executive Summary · Big Success with Big Data 3 Big success with big data Big data is clearly delivering significant value to users who have actually

Met Big bezig? DATA and Analytics Presentati… · 2 Presentatie Big Data: BIG challenge, BIG threat, BIG value or…. BIG hype? © AllRightsReserved2016 (BIG) Data…… What is

Oracle Big Data Science Oracle OpenWorld 2016vlamiscdn.com/papers/Oracle Big Data Science.pdf · Oracle Portfolio of Big Data Science Products Big Data Discovery Big Data SQL Oracle

Introduction to Big Data - scholar.harvard.edu€¦ · Introduction to Big Data Chapter 1 & 2 (Week 1) Course overview & introduction DCCS208(02) Korea University 2019 Fall Asst.

Big Data Visualization: Turning Big Data into Big Insights · PDF fileWhite Paper Big Data Visualization: Turning Big Data Into Big Insights The Rise of Visualization-based Data Discovery

2016 Big Data For Beginners Understanding SMART Big Data, Data Mining & Data Analytics2016 Big Data for Beginners Understanding SMART Big Data, Data Mining & Data Analytics

Big Data Visualization: Turning Big Data Into Big Insights – White ...

2.3 Methods for Big Data What is “Big Data”? Summarizing Big Data.

Introduction to Big Data - Harvard University · Introduction to Big Data Chapter 7 & 8 (Week 4) Data preprocessing. DCCS208(02) Korea University 2019 Fall. Asst. Prof. Minseok Seo.