Big Data
-
Upload
putchong-uthayopas -
Category
Education
-
view
1.687 -
download
1
description
Transcript of Big Data
Introduction to Big Data
Dr. Putchong Uthayopas
Department of Computer Engineering, Faculty of Engineering, Kasetsart University.
Agenda
• Introduction and Motivation
• Big Data Characteristics
• Big Data Technology
• Using Big Data
• Trends
Introduction and Motivation
We are living in the world of Data
Geophysical Exploration
Medical Imaging
Video Surveillance
Mobile Sensors
Gene Sequencing
Smart Grids
Social Media
Some Data sizes ~40 109 Web pages at ~300 kilobytes each = 10 Petabytes
Youtube 48 hours video uploaded per minute;
in 2 months in 2010, uploaded more than total NBC ABC CBS
~2.5 petabytes per year uploaded?
LHC 15 petabytes per year
Radiology 69 petabytes per year
Square Kilometer Array Telescope will be 100 terabits/second
Earth Observation becoming ~4 petabytes per year
Earthquake Science – few terabytes total today
PolarGrid – 100’s terabytes/year
Exascale simulation data dumps – terabytes/second
5
http://www.touchagency.com/free-twitter-infographic/
Information as an Asset
• Cloud will enable larger and larger data to be easily collected and used
• People will deposit information into the cloud – Bank, personal ware house
• New technology will emerge – Larger and scalable storage technology
– Innovative and complex data analysis/visualization for multimedia data
– Security technology to ensure privacy
• Cloud will be mankind intelligent and memory!
“Data is the new oil.”
Andreas Weigend, Stanford (ex Amazon)
Data is more like soup – its
messy and you don’t know
what’s in it….
The Coming of Data Deluge
• In the past, most scientific disciplines could be described as small data, or evendata poor. Most experiments or studies had to contend with just a few hundred or a few thousand data points.
• Now, thanks to massively complex new instruments and simulators, many disciplines are generating correspondingly massive data sets that are described as big data, or data rich. – Consider the Large Hadron Collider, which will eventually generate
about 15 petabytes of data per year. A petabyte is about a million gigabytes, so that qualifies as a full-fledged data deluge.
The Coming Data Deluge: As science becomes more data intensive, so does our language
BY PAUL MCFEDRIES / IEEE SPECTURM FEBRUARY 2011
Particle
physics
data
“Herculean” and
“Heroic”
Scale: an explosion of data
http://www.phgfoundation.org/reports/10364/
“A single sequencer can now generate in a day what it took 10 years to collect for the Human Genome Project”
Creating a connectome
• neuroscientists have set the goal of creating a connectome, a complete map of the brain's neural circuitry. – an image of a cubic millimeter chunk of the brain would comprise
about 1 petabyte of data (at a 5-nanometer resolution).
– There are about a million cubic millimeters of neural matter to map, making a total of about a thousand exabytes (an exabyte is about a thousand petabytes)
– qualifies as what Jim Gray once called an exaflood of data.
The Coming Data Deluge: As science becomes more data intensive, so does our language
BY PAUL MCFEDRIES / IEEE SPECTURM FEBRUARY 2011
The new model is for the data to be captured by
instruments or generated by simulations before
being processed by software and for the resulting
information or knowledge to be stored in computers.
Scientists only get to look at their data fairly late in
this pipeline. The techniques and technologies for
such data-intensive science are so different that it is
worth distinguishing data-intensive science from
computational science as a new, fourth paradigm for
scientific exploration.
—Jim Gray, computer scientist
• The White House today announced a $200 million big-data initiative to create tools to improve scientific research by making sense of the huge amounts of data now available..
• Grants and research programs are geared at improving the core technologies around managing and processing big data sets, speeding up scientific research with big data, and encouraging universities to train more data scientists and engineers.
• The emergent field of data science is changing the direction and speed of scientific research by letting people fine-tune their inquiries by tapping into giant data sets.
• Medical research, for example, is moving from broad-based treatments to highly targeted pharmaceutical testing for a segment of the population or people with specific genetic markers.
So, what is big data?
Big Data
“Big data is data that exceeds the processing
capacity of conventional database systems. The
data is too big, moves too fast, or doesn’t fit the
strictures of your database architectures. To gain
value from this data, you must choose an
alternative way to process it.”
Reference: “What is big data? An introduction to the big data
landscape.”, Edd Dumbill, http://radar.oreilly.com/2012/01/what-is-big-
data.html
Amazon View of Big Data
'Big data' refers to a collection of tools, techniques and technologies which make it easy to work with data at any scale. These distributed, scalable tools provide flexible programming models to navigate
and explore data of any shape and size, from a variety of sources.
The Value of Big Data
• Analytical use – Big data analytics can reveal insights hidden
previously by data too costly to process. • peer influence among customers, revealed by analyzing
shoppers’ transactions, social and geographical data.
– Being able to process every item of data in reasonable time removes the troublesome need for sampling and promotes an investigative approach to data.
• Enabling new products. – Facebook has been able to craft a highly personalized
user experience and create a new kind of advertising business
Big Data Characteristics
3 Characteristics of Big Data
• Volumes of data are larger than those conventional relational database infrastructures can cope with Volume
• Rate at which data flows in is much faster.
• Mobile event and interaction by users.
• Video, image , audio from users Velocity
• the source data is diverse, and doesn’t fall into neat relational structures eg. text from social networks, image data, a raw feed directly from a sensor source.
Variety
Big Data Challenge
Volume
• How to process data so big that can not be move, or store.
Velocity
• A lot of data coming very fast so it can not be stored such as Web usage log , Internet, mobile messages. Stream processing is needed to filter unused data or extract some knowledge real-time.
Variety
• So many type of unstructured data format making conventional database useless.
Big Data Technology
From “Data Driven Discovery in Science:The Fourth Paradigm”, Alex Szalay, Johns Hopkins University
What is needed for big data
• Your data
• Storage infrastructure
• Computing infrastructure
• Middleware to handle BIG Data
• Data Analysis
– Statistical analysis
– Data Mining
• People
How to deal with big data
• Integration of – Storage – Processing – Analysis Algorithm – Visualization
Massive Data
Stream
Stream processing
Processing
Processing
Processing
Visualize
Analysis Storage
How can we store and process massive data
• Beyond capability of a single server
• Basic Infrastructure
– Cluster of servers
– High speed interconnected
– High speed storage cluster
• Incoming data will be spread across the server farm
• Processing is quickly distributed to the farm
• Result is collected and send back
NoSQL (Not Only SQL)
• Next Generation Databases mostly addressing some of the points:
– being non-relational, distributed, open-source and horizontally scalable.
– Used to handle a huge amount of data
– The original intention has been modern web-scale databases.
Reference: http://nosql-database.org/
MongoDB
• MongoDB is a general purpose, open-source database.
• MongoDB features: – Document data model with dynamic
schemas
– Full, flexible index support and rich queries
– Auto-Sharding for horizontal scalability
– Built-in replication for high availability
– Text search
– Advanced security
– Aggregation Framework and MapReduce
– Large media storage with GridFS
What is Hadoop?
- Hadoop or Apache Hadoop
- open-source software framework
- supports data-intensive distributed applications.
- develop by the Apache
- derived from Google's MapReduce and Google File
System (GFS) papers.
- Implement with Java
Overview
master
worker
HDFS
Google Cloud Platform
• App engines – mobile and web app
• Cloud SQL – MySQL on the cloud
• Cloud Storage – Data storage
• Big Query – Data analysis
• Google Compute Engine – Processing of large data
Amazon
• Amazon EC2
– Computation Service using VM
• Amazon DynamoDB
– Large scalable NoSQL databased
– Fully distributed shared nothing architecture
• Amazon Elastic MapReduce (Amazon EMR)
– Hadoop based analysis engine
– Can be used to analyse data from DynamoDB
Issues
• I/O capability of a single computer is limited , how to handle massive data
• Big Data can not be moved
– Careful planning must be done to handle big data
– Processing capability must be there from the start
Using Big Data
WHAT FACEBOOK KNOWS
http://www.facebook.com/data
Cameron Marlow calls himself Facebook's "in-
house sociologist." He and his team can
analyze essentially all the information the site
gathers.
Study of Human Society
• Facebook, in collaboration with the University of Milan, conducted experiment that involved
– the entire social network as of May 2011
– more than 10 percent of the world's population.
• Analyzing the 69 billion friend connections among those 721 million people showed that
– four intermediary friends are usually enough to introduce anyone to a random stranger.
The links of Love
• Often young women specify that they are “in a relationship” with their “best friend forever”.
– Roughly 20% of all relationships for the 15-and-under crowd are between girls.
– This number dips to 15% for 18-year-olds and is just 7% for 25-year-olds.
• Anonymous US users who were over 18 at the start of the relationship
– the average of the shortest number of steps to get from any one U.S. user to any other individual is 16.7.
– This is much higher than the 4.74 steps you’d need to go from any Facebook user to another through friendship, as opposed to romantic, ties.
http://www.facebook.com/notes/facebook-data-team/the-links-of-
love/10150572088343859
Graph shown the relationship of anonymous US users who were
over 18 at the start of the relationship.
Why?
• Facebook can improve users experience – make useful predictions about users' behavior
– make better guesses about which ads you might be more or less open to at any given time
• Right before Valentine's Day this year a blog post from the Data Science Team listed the songs most popular with people who had recently signaled on Facebook that they had entered or left a relationship
How facebook handle Big Data?
• Facebook built its data storage system using open-source software called Hadoop. – Hadoop spreading them across many machines inside a
data center. – Use Hive, open-source that acts as a translation service,
making it possible to query vast Hadoop data stores using relatively simple code.
• Much of Facebook's data resides in one Hadoop store more than 100 petabytes (a million gigabytes) in size, says Sameet Agarwal, a director of engineering at Facebook who works on data infrastructure, and the quantity is growing exponentially. "Over the last few years we have more than doubled in size every year,”
Google Flu
• pattern emerges when all the flu-related search queries are added together.
• We compared our query counts with traditional flu surveillance systems and found that many search queries tend to be popular exactly when flu season is happening.
• By counting how often we see these search queries, we can estimate how much flu is circulating in different countries and regions around the world.
http://www.google.org/flutrends/abo
ut/how.html
From “Data Driven Discovery in Science:The Fourth Paradigm”, Alex Szalay, Johns Hopkins University
From “Data Driven Discovery in Science:The Fourth Paradigm”, Alex Szalay, Johns Hopkins University
Preparing for BigData • Understanding and preparing your data
– To effectively analyse and, more importantly, cross-analyse your data sets – this is often where the most insightful results come from – you need to have a rigorous knowledge of what data you have.
• Getting staff up to scratch – Finding people with data analysis experience
• Defining the business objectives – Once the end goal has been decided then a strategy can be created for implementing big data analytics
to support the delivery of this goal
• Sourcing the right suppliers and technology – But in terms of storage, hardware, and data warehousing, you will need to make a range of decisions to
make sure you have all the capabilities and functionality required to meet your big data needs.
http://www.thebigdatainsightgroup.com/site/article/preparing-big-data-revolution
Trends
Trends
• A move toward large and scalable Virtual Infrastructure – Providing computing service
– Providing basic storage service
– Providing Scalable large database • NOSQL
– Providing Analysis Service
• All these services has to come together – Big data can not moved!
Issues
• Security – Will you let an important data being accumulate outside your
organization? • If it is not an important data, why analyze them ?
– Who own the data? If you discontinue the service, is the data being destroy properly.
– Protection in multi-tenant environment
• Big data can not be moved easily – Processing have to be near. Just can not ship data around
• So you finally have to select the same cloud for your processing. Is it available, easy, fast?
• New learning, development cost – Need new programming, porting? – Tools is mature enough?
When to use Big data on the Cloud
• When data is already on the cloud – Virtual organization – Cloud based SaaS Service
• For startup – CAPEX to OPEX – No need to maintain large infra – Focus on scalability and pay as you go – Data is on the cloud anyway
• For experimental project – Pilot for new services
Summary
• Big data is coming.
– Changing the way we do science
– Big data are being accumulate anyway
– Knowledge is power.
• Better understand your customer so you can offer better service
• Tools and Technology is available
– Still being developed fast
Thank you