Object-Oriented Databases Object Oriented Databases - ODBMS.org
Introduction to · 2012-07-17 · Introduction to András Garzó ([email protected]) ... HBase...
Transcript of Introduction to · 2012-07-17 · Introduction to András Garzó ([email protected]) ... HBase...
![Page 2: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/2.jpg)
Google BigTable (2006)
● Distributed multi level map
● Fault tolerant
● Horizontal Scalability
● Runs on commodity hardware
● Self managing
● Large number of R/W
● Tight integration with MapReduce
![Page 3: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/3.jpg)
HBase
● Open source BigTable implementation
● Based on Hadoop stack
● Data stored on HDFS
● Integrated with MapReduce
![Page 4: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/4.jpg)
HBase Data Model
![Page 5: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/5.jpg)
Column-oriented database
HBase column oriented only in column-family level!
![Page 6: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/6.jpg)
HBase Architecture
![Page 7: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/7.jpg)
Auto Sharding
![Page 8: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/8.jpg)
Distribution
![Page 9: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/9.jpg)
Auto Sharding and Distribution
● Region is the unit of scalability
● Sorted, contiguous range of rows
● Load balancing and failover
● Split automatically or manually to scale with
growing data
● Capacity is solely a factor of cluster nodes vs.
region per node
![Page 10: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/10.jpg)
Regions and Splitting
![Page 11: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/11.jpg)
Regions and Splitting
![Page 12: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/12.jpg)
Regions and Splitting
![Page 13: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/13.jpg)
HMaster
● Coordinates region splitting
● Load balancing
● Table management
● Multiple masters for failover
![Page 14: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/14.jpg)
Zookeeper
● Master election
● Locate –ROOT- region
● Region Server membership
![Page 15: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/15.jpg)
Where is my row?
![Page 16: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/16.jpg)
Where is my row?
![Page 17: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/17.jpg)
Where is my row?
![Page 18: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/18.jpg)
Where is my row?
![Page 19: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/19.jpg)
Where is my row?
![Page 20: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/20.jpg)
Inside the Region
![Page 21: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/21.jpg)
Writing to HBase
![Page 22: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/22.jpg)
Writing to HBase
![Page 23: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/23.jpg)
Writing to HBase
![Page 24: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/24.jpg)
Writing to HBase
![Page 25: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/25.jpg)
Writing to HBase
![Page 26: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/26.jpg)
Writing to HBase
![Page 27: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/27.jpg)
Reading from HBase
![Page 28: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/28.jpg)
Reading from HBase
![Page 29: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/29.jpg)
Merge Reads
![Page 30: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/30.jpg)
Logical and physical layout
![Page 31: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/31.jpg)
Logical and physical layout
● Logical layout does not match physical one
● All values stored with the full coordinates,
including Row Key, Column Family, Column
Qualifier and Timestamp
● Folds columns into „row per column”
● NULLs are cost free
● Versions are multiple „rows” in folded table
![Page 32: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/32.jpg)
Key Cardinality
![Page 33: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/33.jpg)
Key Cardinality
● Best performance is gained from using row
keys
● Selecting column families reduces the amount
of the data to be scanned
● Time range bound reads can skip store files
(and Bloom Filters too!)
● Pure value based filtering is a full table
scan!
![Page 34: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/34.jpg)
Tall-Narrow or Flat-Wide tables?
● Same storage footprint
● Atomicity only on row level
● Rows do not split
● Put more details into the row keys
● Tall with scans, wide with gets
![Page 35: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/35.jpg)
Example
![Page 36: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/36.jpg)
Sequential read and write
![Page 37: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/37.jpg)
HBase and MapReduce
How data locality achieved?
● Region server and data node runs on the same
node
● HBase shuts down very rarely
● DataNodes help: when Region Servers write to
HDFS, data blocks will be stored locally also
![Page 38: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/38.jpg)
HBase vs. RDBMS
HBase RDBMS
Column oriented Row oriented
Flexible scheme, add columns on the fly Fixed scheme
Good with sparse table Not optimized for sparse tables
No query language (just Scan and Get) SQL
No joins (but we can do with MapReduce) Optimized for joins
Horizontal scalability Hard to scale
No transactions Transactional
Consistent Consistent
It’s a really wrong comparison!
![Page 39: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/39.jpg)
HBase on the CAP triangle
![Page 40: Introduction to · 2012-07-17 · Introduction to András Garzó (garzo@ilab.sztaki.hu) ... HBase RDBMS Column oriented Row oriented Flexible scheme, add columns on the fly Fixed](https://reader036.fdocuments.us/reader036/viewer/2022062506/5f0ba5ee7e708231d4318954/html5/thumbnails/40.jpg)
Q&A