Efficient Transaction Processing in SAP HANA Database.pdf
-
Upload
prasenjit-roy -
Category
Documents
-
view
227 -
download
0
Transcript of Efficient Transaction Processing in SAP HANA Database.pdf
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 1/31
Efficient Transactions Processing in SAP HANADatabase - The End of a Column Store Myth
Vishal Sikka, Franz Farber, Wolfgang Lehner, Sang Kyun Cha,Thomas Peh, Christof Bornhovd
presented by Cong Guo
February 10, 2015
Cong Guo Efficient Transactions Processing in SAP HANA Database
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 2/31
Outline
Motivation
Architecture of SAP HANA
Lifecycle Management of Database RecordsMerge Optimization
Conclusion
Discussion
Cong Guo Efficient Transactions Processing in SAP HANA Database
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 3/31
Motivation
Usage perspective -various types of workloads and usagepatterns
OLTP - high concurrency, frequent updates, and selectivepoint queries
OLAP - long transactions, infrequent updates, aggregationqueries, and historical data
Zoo of specialized systems
Complex and error-proneHigh total cost of ownership (TCO)Used for performance
Cong Guo Efficient Transactions Processing in SAP HANA Database
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 4/31
SAP HANA Appliance At a Glance
Replace the zoo of specialized systems with a flexible platform
Cong Guo Efficient Transactions Processing in SAP HANA Database
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 5/31
Features of SAP HANA database
Has a girl’s name (Hanna)
Comprises multiple engines from relational data to graphs tounstructured text data
Supports application-specific business objects and logicdirectly
Communicates with the application layer efficiently
Supports efficient processing for both OLTP and OLAPworkloads
Cong Guo Efficient Transactions Processing in SAP HANA Database
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 6/31
Outline
Motivation
Architecture of SAP HANA
Lifecycle Management of Database RecordsMerge Optimization
Conclusion
Discussion
Cong Guo Efficient Transactions Processing in SAP HANA Database
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 7/31
Architecture of SAP HANA
Cong Guo Efficient Transactions Processing in SAP HANA Database
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 8/31
Architecture of SAP HANA
Cong Guo Efficient Transactions Processing in SAP HANA Database
C G
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 9/31
Calculation Graph Model
An internal representation of query is mapped to a CalculationGraph
Source nodes - table structures or outcome of other calcgraphs
Inner nodes - logical operators
Operators
Intrinsic operators like projection, joins, union etc
Business algorithms like currency conversionDynamic SQL nodes, custom nodes, R nodes, and L nodesSplit and combine
Cong Guo Efficient Transactions Processing in SAP HANA Database
C l l i G h M d l E l
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 10/31
Calculation Graph Model - Example
Cong Guo Efficient Transactions Processing in SAP HANA Database
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 11/31
O tli
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 12/31
Outline
Motivation
Architecture of SAP HANA
Lifecycle Management of Database RecordsMerge Optimization
Conclusion
Discussion
Cong Guo Efficient Transactions Processing in SAP HANA Database
Lif l M t f D t b R d
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 13/31
Lifecycle Management of Database Records
Cong Guo Efficient Transactions Processing in SAP HANA Database
L1 d lt St
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 14/31
L1-delta Storage
Accepts all incoming data requests
Stores records in row format (write-optimized)
No data compression
Holds 10,000 to 100,000 rows per single-node
Cong Guo Efficient Transactions Processing in SAP HANA Database
L2 delta Storage
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 15/31
L2-delta Storage
Accepts bulk inserts
Stores records in column format (an index vector)
Uses dictionary encoding for better memory usageUnsorted dictionaryCSB-Tree based secondary index for point access
Inverted index mapping value IDs to positions
Cong Guo Efficient Transactions Processing in SAP HANA Database
Main Store
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 16/31
Main Store
Stores records in column format
Employs a sorted dictionary
Highest compression rate
Positions in the dictionary are stored in a bit-packed mannerDictionary is also compressed using RLE and other techniques
Cong Guo Efficient Transactions Processing in SAP HANA Database
Unified Table Access
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 17/31
Unified Table Access
A common abstract interface to access different stores
Records are propagated asynchronously
Two transformations between stores called merge steps
Cong Guo Efficient Transactions Processing in SAP HANA Database
Merge from L1 delta to L2 delta
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 18/31
Merge from L1-delta to L2-delta
Row format to column format conversion
Merge Steps
Appending new entries to the dictionary (in parallel)Storing column values using the dictionary encodings (inparallel)Removing propagated entries from the L1-delta
Cong Guo Efficient Transactions Processing in SAP HANA Database
Merge from L1-delta to L2-delta
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 19/31
Merge from L1-delta to L2-delta
A straightforward task
The first two steps can be performed in parallel
L2-delta data structures are not reconstructed
Incremental merge
Minimally invasive to running transactions
Cong Guo Efficient Transactions Processing in SAP HANA Database
Merge from L2-delta to Main
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 20/31
Merge from L2 delta to Main
A resource intensive task
The old L2-delta is closed for updates
A new empty L2-delta is created
A new main structure is created
The merge is retried on failures
Cong Guo Efficient Transactions Processing in SAP HANA Database
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 21/31
Outline
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 22/31
Outline
Motivation
Architecture of SAP HANA
Lifecycle Management of Database Records
Merge Optimization
Conclusion
Discussion
Cong Guo Efficient Transactions Processing in SAP HANA Database
Merge Optimization
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 23/31
Merge Optimization
The classic merge needs optimization
L2-delta to main merge is resource intensiveMain store needs high compression rate
Re-sorting merge: higher compression rate
Partial merge: reduce overhead of merge
Cong Guo Efficient Transactions Processing in SAP HANA Database
Classic Merge
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 24/31
g
Cong Guo Efficient Transactions Processing in SAP HANA Database
Re-Sorting Merge
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 25/31
g g
Individual columns are re-sorted to gain higher compressionrate
A mapping table of row positions is added to reconstruct therow
Sort order of columns are based on statistics from main andL2-delta
Cong Guo Efficient Transactions Processing in SAP HANA Database
Re-Sorting Merge
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 26/31
g g
Cong Guo Efficient Transactions Processing in SAP HANA Database
Partial Merge
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 27/31
g
Reduce merge overhead due to a large table size
Split the main into two independent structures
Passive main- not part of the merge processActive main- takes part in the merge process with the L2-delta- only holds new values not in the passive main
Accesses are resolved in both dictionaries and parallel scansare performed on both structures
Cong Guo Efficient Transactions Processing in SAP HANA Database
Partial Merge
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 28/31
g
Cong Guo Efficient Transactions Processing in SAP HANA Database
Conclusion
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 29/31
The HANA database is
the core of SAP application ecosystem
a main-memory database that efficiently supports both OLTP
and OLAP
consisting of different states of data structures but providing acommon interface
optimized for memory requirements and query processing
Cong Guo Efficient Transactions Processing in SAP HANA Database
Summary
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 30/31
Cong Guo Efficient Transactions Processing in SAP HANA Database
Discussion
7/24/2019 Efficient Transaction Processing in SAP HANA Database.pdf
http://slidepdf.com/reader/full/efficient-transaction-processing-in-sap-hana-databasepdf 31/31
How does HANA determine when to merge the storages?
Currently based on data sizeL2-delta is used to soften the problem
Differences between main-memory and disk based DBMSs
Cache performance mattersThe complexity of buffer pool management is reducedPersistency is more challenging
Differences between column stores and row stores
Compression
Cong Guo Efficient Transactions Processing in SAP HANA Database