Chapter 1: Introduction, Exploring Data

. . . . . .

IntroductionTerminologies Identi ication

Summary StatisticsBasic Visualization

Visualization for High-Dimensional DataOLAP and Multidimensional Data Analysis

Supplementary

.

...... Chapter 1: Introduction, Exploring Data

Richard LiuSchool of Mathematics, XMU

February 26, 2020

Richard Liu Chapter 1: Introduction, Exploring Data 1 / 61

. . . . . .




Supplementary

.. Content...1 Introduction...2 Terminologies Identi ication...3 Summary Statistics...4 Basic Visualization...5 Visualization for High-Dimensional Data...6 OLAP and Multidimensional Data Analysis...7 Supplementary


. . . . . .




Supplementary

Section 1

Introduction


. . . . . .




Supplementary

.. IntroductionFour topics are discussed in this chapter.

Terminologies Identi icationSummary StatisticsVisualizationOn-Line Analytical Processing(OLAP)

Used for exploring multidimensional arrays of values.This chapter is tightly related to the area known asExploratory Data Analysis (EDA), other parts of which areCluster Analysis and Anomaly Detection which are coveredin Chapter 8 to 10.


. . . . . .




Supplementary

.. Terminologies Identi ication

Data ScienceMachine LearningData MiningBusiness Analytics


. . . . . .




Supplementary


Classi icationRegressionClustering


. . . . . .




Supplementary


Numerical DataCategorical DataBinary Data


. . . . . .




Supplementary


Low-dimensional DataHigh-dimensional Data


. . . . . .




Supplementary

.. Machine Learning Structures

Training SetTest SetValidation Set


. . . . . .




Supplementary

So, what is data science?


. . . . . .




Supplementary

Section 3

Summary Statistics


. . . . . .




Supplementary

.. Summary StatisticsA quantity which describes an overall characteristic of oneset of values..De inition 1: Frequency..

......

Given a random variable x, which can take values{v1, · · · , vn} (vi ̸= vj for i ̸= j), then

Frequency(vi) =#{vi}n

where #{vi}means the number of value vi in a speci ic dataset S.


. . . . . .




Supplementary

.. Summary Statistics

.De inition 2: Mode..

......Given a data set S, Mode(S) = argmaxvi Frequency(vi)

For the data in reality, usually mode happens more thanonce.Frequently used as an indicator of the missingvalue.(Why?)


. . . . . .




Supplementary

.. Summary Statistics.De inition 3: Percentiles..

......The pth percentile xp is de ined as a value x such that p% ofthe observed values of x are less than xp.

Obviously x0% = min(x), x100% = max(x).De inition 4: Mean..

......

Assume {x1, · · · , xm} is an ordered set of observed values,denoted by x, then

mean(x) = x̄ =1

m

m∑i=1

xi


. . . . . .




Supplementary

.. Summary Statistics.Notation 1..

......De ine x↓ = {x(1), · · · , x(m)} a permutation of x,x(1) ≥ x(2) ≥ · · · ≥ x(m)..De inition 5: Median..

......

median(x) = median(x↓) ={x(r+1) m = 2r+ 112(x(r) + x(r+1)) m = 2r

wherem = |x|, the cardinality of set x.

Mean and median are the measures of the location of a set ofvalues. Richard Liu Chapter 1: Introduction, Exploring Data 15 / 61

. . . . . .




Supplementary

.. Summary Statistics.De inition 6: Range........ range(x) = max(x)−min(x).De inition 7: Variance..

......variance(x) = s2x =

1

m− 1

m∑i=1

(xi − x̄)2

.De inition 8: Standard Deviation..

...... sd(x) = sx =√

s2x

There three are the measures of the spread of a set of values.Richard Liu Chapter 1: Introduction, Exploring Data 16 / 61

. . . . . .




Supplementary

.. Other Quantities

Due to the existence of outliers, previous statistics are notrobust, some alternatives are considered therefore..De inition 9: Absolute Average Deviation(AAD)..

......AAD(x) = 1

m

m∑i=1

|xi − x̄|

.De inition 10: Median Absolute Deviation(MAD)..

...... MAD(x) = median({|xi − x̄|}mi=1)


. . . . . .




Supplementary

.. Other Quantities

.De inition 11: Interquartile Range(IQR)..

...... IQR(x) = x75% − x25%


. . . . . .




Supplementary

.. Multivariate CaseSometimes a data set consists of several sets of valuesbelonging to different attributes of that. In other words,

x = (x1, · · · , xn)

where n is the number of attributes, xi is a set of values,comprising the ith attribute of all observed data.In this case, every observed data point is a vector, not anumber. So we de ine mean as

x̄ = (x̄1, · · · , x̄n)


. . . . . .




Supplementary

.. Multivariate Case.De inition 12: Covariance..

......covariance(xi, xj) =

1

m− 1

m∑k=1

(xki − x̄i)(xkj − x̄j)

.De inition 13: Correlation..

......rij = correlation(xi, xj) =

covariance(xi, xj)sisj

Evidently correlation(xi, xj) = variance(xi)


. . . . . .




Supplementary

Section 4

Basic Visualization


. . . . . .




Supplementary

.. Motivation

People could absorb large amounts of data from onegraph quickly.Make use of the domain knowledge that is 'locked up inpeople's heads.'(Hard for data mining)


. . . . . .




Supplementary

.. One example


. . . . . .




Supplementary

.. General ConceptsData objects, attributes and the relationships amongdat a objects are translated into graphical elementssuch as points, lines, shapes and colors.Note that the representation depends on the type ofattribute(nominal, ordinal, continuous). When thevalue itself has order it is OK to represent them into acoordinate system (with x, y, z axes).You need to preserve some important informationabout relative attributes(such as physical location)People would like to believe that data points that arevisually close to each other have similar values for theirattributes. Richard Liu Chapter 1: Introduction, Exploring Data 24 / 61

. . . . . .




Supplementary

.. Challenge

How to be easily observable? This is what visualization paysattention to most.


. . . . . .




Supplementary

.. Data Arrangement


. . . . . .




Supplementary

.. Selection

Sometimes it is extremely hard to show all data objectsand attribute on one graph. So we use selection.Example: Many attributes -> a series oftwo-dimensional plots.


. . . . . .




Supplementary

.. Visualization Techniques

Stem and Leaf Plots

Splitting values into groups, each containing thosevalues that are the same except for the last digit.Suitable for small values.

Histograms

Displaying the distribution of values for attributes bydividing the possible values into bins and showing thenumber of objects that fall into each bin.


. . . . . .




Supplementary

.. Examples


. . . . . .




Supplementary


Box Plot

Show the distribution of the values of a singlenumberical attribute.

Pie Chart

Similar to histogram, typically used for dividingcategorical attributes.Hard for distinguishing. Not preferred in technicalwork.


. . . . . .




Supplementary

.. Examples


. . . . . .




Supplementary


Scatter PlotWidely used for judging the relation between twoattributes given a series of data objects.


. . . . . .




Supplementary

.. Examples


. . . . . .




Supplementary


Percentile PlotsEmpirical Cumulative Distribution Functions(ECDF)

A function graph.For any given x, it shows the fraction(or, probability) ofthe points that are less than x.

.Notation 2..

......F(x) = P(X ≤ x), where X is a random variable.


. . . . . .




Supplementary

.. Example


. . . . . .




Supplementary

..One special case: Demonstrating severalattributes in one graph


. . . . . .




Supplementary

.. Visualization Techniques (Details Removed)

Contour PlotsSurface PlotsVector Field PlotsLower-Dimensional SlicesAnimation


. . . . . .




Supplementary

Section 5

Visualization for High-DimensionalData


. . . . . .




Supplementary


MatrixCorrelation Matrix: One important step for dataanalysis, in order to capture the features of the wholedata set.A simple kind of Clustering.

Parallel CoordinatesOne coordinate axis for each attribute but the differentaxes are parallel to one other instead of perpendicular.Also, sometimes could be used to ind features.


. . . . . .




Supplementary

.. Examples


. . . . . .




Supplementary

Section 6

OLAP and Multidimensional DataAnalysis


. . . . . .




Supplementary

.. Discretizing

Discretizing continuous attribute into categoricalattribute.In this way we could arrange the data within a table ora multidimensional data representation.Cross Tabulation can also be implemented after doingso.


. . . . . .




Supplementary

.. Examples


. . . . . .




Supplementary

.. Cross Tabulation


. . . . . .




Supplementary

.. Aggregation

Aggregation is one of the most general method foranalyzing multidimensional data objects.One example is summation.


. . . . . .




Supplementary

.. Examples


. . . . . .




Supplementary

.. Dimensionality Reduction

In descriptive data analysis, this means trying to reducethe number of attributes shown in the table/graph withaggregation.In regression, this means trying to select fewerresponses in order to diminish the correlation amongthem.Some examples: PCA, SVD.


. . . . . .




Supplementary

.. These subjects are ignored

PivotingSlicing and DicingRoll-UpDrill-Down


. . . . . .




Supplementary

Section 7

Supplementary


. . . . . .




Supplementary

.. Preliminary

Sample SpaceProbabilityDistribution FunctionExpectationVarianceCentral Limit Theorem


. . . . . .




Supplementary

.. Propositions in Mathematical StatisticsAssume x1, · · · , xn ∼ N(µ, σ2), then.Proposition 1..

......

Let x̄ = 1n∑n

i=1 xi, then

E(x̄) = µ

.Proposition 2..

......

Let s2 = 1n−1

∑ni=1(xi − x̄)2, then

E(s2) = σ2


. . . . . .




Supplementary

.. Propositions in Mathematical Statistics.Proposition 3..

......

Let f(c) = ∑ni=1(xi − c)2, then

x̄ = argminc

f(c)

.Proposition 4..

......

Let f(c) = ∑ni=1 |xi − c|, then

median(x) = argminc

f(c)


. . . . . .




Supplementary

.. Python in Numerical Computation

For more information, see:https://weakcha.github.io/WISERCLUB-Final/Experiments.html


https://weakcha.github.io/WISERCLUB-Final/Experiments.html

https://weakcha.github.io/WISERCLUB-Final/Experiments.html

. . . . . .




Supplementary

Thank you!


Chapter 1: Introduction, Exploring Data

Documents

Transcript of Chapter 1: Introduction, Exploring Data