Introduction to Big Data - Harvard University€¦ · Introduction to Big Data Chapter 5 & 6 (Week...

Introductionto Big Data

Chapter 5 & 6 (Week 3)Data Structure & Data types

DCCS208(02) Korea University 2019 Fall

Asst. Prof. Minseok [email protected]

Contents

Record-based data

Types of data2.

Data structure

Data Structure1. Discrete & Continuous Attribute

Tabulation

Structured & unstructured data

Graph-based data

Ordered data

01Data StructureIntroduction to Big Data

copyrightⓒ 2018 All rights reserved by Korea University 4

Data StructureWhat is dataset?

What is dataset?

Tid Refund Marital Status

Taxable Income Cheat

1 Yes Single 125K No

2 No Married 100K No

3 No Single 70K No

4 Yes Married 120K No

5 No Divorced 95K Yes

6 No Married 60K No

7 Yes Divorced 220K No

8 No Single 85K Yes

9 No Married 75K No

10 No Single 90K Yes 10

Attributes, Chracteristics, Features, Variables, ..., etc.

Object

Instance

Individual

Sample

Subject

...

• Set of objects consisting oftheir attributes.


Data StructureWhat is dataset?

What is variables?Attributes, Chracteristics, Features, ..., etc.

• Properties / features of specific object.

• i.e. Human• Eye size (mm unit)• Eye color• Skin color• Height (cm unit)• Wear glasses or not• Gender• Age• Length of finger (cm unit)• ...


Discrete & Continuous AttributeTypes of variable

Continuous Attribute

• It has a real number a s property value

• A continuous variable is one which can take on infinitely many,uncountable values.

• Continuous attributes are usually represented as floating pointvariables


In Korean...?


Discrete & Continuous AttributeTypes of variable

Discrete Attribute

• Finite or infinite set of countable values• Usually expressed as an integer variable• Binary attribute is a special form of discrete attribute• Continuous variables can also be convereted to discrete variables

thorugh binning


TabulationImportance of Tabulation

It is a systematic and logical arrangement of classified data in rows and columns.

Let’s think about how we can collect structured or unstructured data


Collection of the dataTwo types of processes

Usually, this is data-science process

Experimental design

Determination of population Set variables Get values

Experimental design

Determination of population Get values Prescreening

Usually, we mentioned that ‘Data Mining’

Tabulation process

Tabulation process


Let’s try tabulationPractice of tabulation

AAAAA AAAAA AAAAA AAAAABB BB BB BBCCC CCC CCC CCCD D D D

Try to tabulize above unstructured data


Features of ‘unstructured” dataImportance of tabulation

It does not reside in traditional databases and data warehouses.

It may have an internal structure, but does not fit a relational data model.

It generated by both humans and machines.

It will usually be textual and multimedia content, Machine-to-machinecommunication

Examples include• Personal messaging – email, instant messages, tweets, chat• Business documents – business reports, presentations, survey

responses• Web content – web pages, blogs, wikis, audio files, photos, videos• Sensor output – satellite imagery, geolocation data, scanner

transactions


Current dataZettabytes’ unstructured data

About 85% is unstructured data


Big DataImportance of Tabulation in Big Data era

Data sets of such size, complexity and volatility that their businessvalue cannot be fully realised with existing data capture, storage,processing, analysis and management capabilities

The systematic use ofunstructured data is aBig Data challenge!

02Types of DataIntroduction to Big Data


Types of DatasetData... data... data.......

Record-based data• Data matrix• Document data• Transaction data• ...

Graph-based data• World wide web• Molecular structure• Map data• ...

Ordered data• Spatial data• Temporal data• Sequential data• Genetic/Genomic sequence data


Types of DatasetRecord-based data

Data that consists of a collection of records, each of which consistsof a fixed set of attributes


Types of DatasetData matrix

This is a term derived from linear algebra.

When composed of a fixed number of numerical attributes, anobject (record) can be considered as a point in multidimensionalspace.

Such data is represented by n x p matrices, where n rows eachrepresent an object and p columns each represent an attribute.


Types of DatasetDocument data

Each document can be represented by a term vector.

Each term corresponds to a component of the vector.

Each value corresponds to the number of times the term appearedin the document.


Types of DatasetTransaction Data

As a special type of record, each record (transaction) is acollection of items.

It also known as ‘market basket data’.


Types of DatasetGraph Data

G = (V, E)• V is set of vertices (nodes)• E is set of edges (arcs or link)



Set of HTML documents



Chemical structure data

Social network data


Types of DatasetOrdered data

Sequences of transactions



Genetic / Genomics sequence data



Time-series data



Spatio- Temporal data

End of Slide


Topic of next classComing soon...

Introduction to R programmingThursday

Data Preprocessing Quality control Similarity & dissimilarity Distance metric Fundamental statistics

Next Wednesday

Introduction to Big Data - Harvard University€¦ · Introduction to Big Data Chapter 5 & 6 (Week...

Documents

Transcript of Introduction to Big Data - Harvard University€¦ · Introduction to Big Data Chapter 5 & 6 (Week...