What is the Need of Big data Technology

when we have robust, high-performing,

relational database management system

?

Data Stored in structured format like PK, Rows,

Columns, Tuples and FK .

It was for just Transactional data analysis.

Later using Data warehouse for offline data.

(Analysis done within Enterprise)

Massive use of Internet and Social Networking(FB,

Linkdin) Data become less structured.

Data is stored on central server.

‘Big Data’ is similar to ‘small data’, but bigger

…but having data bigger it requires different approaches:› Techniques, tools and architecture

…with an aim to solve new problems› …or old problems in a better way

Volume

• Data quantity

Velocity

• Data Speed

Variety

• Data Types

HADOOP

Open-source data storage and processing API

Massively scalable, automatically parallelizable

Based on work from Google

GFS + MapReduce + BigTable

Current Distributions based on Open Source and Vendor Work

Apache Hadoop

Cloudera – CH4 w/ Impala

Hortonworks

MapR

AWS

Windows Azure HDInsight

HDFS

Storage

Self-healing

high-bandwidth

clustered storage

MapReduce

Processing

Fault-tolerant

distributed

processing

HDFS is a file system written in Java

Sits on top of a native file system

Provides redundant storage for massive

amounts of data

Use cheap, unreliable computers

Data is split into blocks and stored on multiple

nodes in the cluster

› Each block is usually 64 MB or 128 MB (conf)

Each block is replicated multiple times (conf)

› Replicas stored on different data nodes

Large files, 100 MB+

Master Nodes Slave Nodes

NameNode

› only 1 per cluster

› metadata server and database

› SecondaryNameNode helps with some housekeeping

• JobTracker

• only 1 per cluster

• job scheduler

DataNodes

› 1-4000 per cluster

› block data storage

• TaskTrackers

• 1-4000 per cluster

• task execution

A single NameNode stores all metadata

Filenames, locations on DataNodes of each

block, owner, group, etc.

All information maintained in RAM for fast lookup

File system metadata size is limited to the amount

of available RAM on the NameNode

DataNodes store file contents

Stored as opaque ‘blocks’ on the underlying filesystem

Different blocks of the same file will be stored on different DataNodes

Same block is stored on three (or more) DataNodes for redundancy

DataNodes send heartbeats to NameNode

› After a period without any heartbeats, a DataNode is assumed to be lost

› NameNode determines which blocks were on the lost node

› NameNode finds other DataNodes with copies of these blocks

› These DataNodes are instructed to copy the blocks to other nodes

› Replication is actively maintained

The Secondary NameNode is not a failover NameNode

Does memory-intensive administrative functions for the NameNode

Should run on a separate machine

datanode daemon

Linux file system

…

tasktracker

slave node

datanode daemon

Linux file system

…

tasktracker

slave node

datanode daemon

Linux file system

…

tasktracker

slave node

namenode

namenode daemon

job submission node

jobtracker

MapReduce

MapReduce

JobTrackerMapReduce job

submitted by

client computer

Master node

TaskTracker

Slave node

Task instance

TaskTracker

Slave node

Task instance

TaskTracker

Slave node

Task instance

In our case: circe.rc.usf.edu

Preparing for

MapReduceLoading

Files

64 MB

128 MB

File System

Native file system

HDFS

Cloud

Output

Immutable

You Define

Input, Map, Reduce, Output

Use Java or other programming

language

Work with key-value pairs

Input: a set of key/value pairs

User supplies two functions:

› map(k,v) list(k1,v1)

› reduce(k1, list(v1)) v2

(k1,v1) is an intermediate key/value pair

Output is the set of (k1,v2) pairs

InputFormat

Map function

Partitioner

Sorting & Merging

Combiner

Shuffling

Merging

Reduce function

OutputFormat

• Task Tracker

• Data Node

• Task Tracker

• Data Node

• Task Tracker

• Data Node

• Name Node

• Job Tracker

Master Node

Slave Node

1

Slave Node

2

Slave Node

3

MapReduce Example -

WordCount

InputFormat:

› TextInputFormat

› KeyValueTextInputFormat

› SequenceFileInputFormat

OutputFormat:

› TextOutputFormat

› SequenceFileOutputFormat

2

8

Probably the most complex aspect of MapReduce!

Map side

› Map outputs are buffered in memory in a circular buffer

› When buffer reaches threshold, contents are “spilled” to disk

› Spills merged in a single, partitioned file (sorted within each

partition): combiner runs here

Reduce side

› First, map outputs are copied over to reducer machine

› “Sort” is a multi-pass merge of map outputs (happens in

memory and on disk): combiner runs here

› Final merge pass goes directly into reducer

Mapper

Reducer

other mappers

other reducers

circular buffer

(in memory)

spills (on disk)

merged spills

(on disk)

intermediate files

(on disk)

Combiner

Combiner

Writable Defines a de/serialization protocol.

Every data type in Hadoop is a Writable.

WritableComprable Defines a sort order. All keys must be

of this type (but not values).

IntWritable

LongWritable

Text

…

Concrete classes for different data types.

SequenceFiles Binary encoded of a sequence of

key/value pairs

Map function

Reduce function

Run this program as a

MapReduce job

Hadoop ClusterYou

1. Load data into HDFS

2. Develop code locally

3. Submit MapReduce job3a. Go back to Step 2

4. Retrieve data from HDFS

Applications for Big Data Analytics

Homeland Security

Finance Smarter HealthcareMulti-channel

sales

Telecom

Manufacturing

Traffic Control

Trading Analytics Fraud and Risk

Log Analysis

Search Quality

Retail: Churn, NBO

CASE STUDY : 1Environment Change Prediction to Assist Formers Using Hadoop

Thank you

Questions ?