HANA & XS Engine Overview

SAP HANA & XS Engine

Overview

Internal

© 2013 SAP AG. All rights reserved. 2 Ramp-Up Knowledge Transfer Customer

Outline

HANA

• Introduction

• Architecture

HANA Studio

• Introduction

Modeled Views

• Attribute, Analytic, Calculation

XS Engine

• Introduction

• Repository

• Architecture

XS Engine Applications

• Application Containers

• Regi and Workspaces

• Deployment

Resources


Introduction to HANA

“SAP HANA allows for the processing of massive quantities of real-time data in main memory to

provide immediate results”

How?

• Multi-core architectures and algorithms allowing for high process parallelization

• In-memory design

• Efficient data storage

(column store, compression, insert only deltas, no more materialized aggregates, …)


Multi-Core and In-Memory

Traditional Databases

• Bottlenecked by disk I/O due to limited RAM

HANA

• With multi-core CPUs and RAM no longer being

limited, bottleneck shifts to data transfer

between CPU cache and main memory


In-Memory Design


• Separate data management and applications into a DB and application layer

• Force data to travel from DB to application layer before it can be analyzed or modified

HANA

• Moves data intensive application logic to the DB layer

• Extends data processing capabilities of the DB

• Avoids layer boundary bottleneck

• Embedding is done via extensions to SQL (ex. SQLScript)


Efficient Data Storage


• Even with data in-memory, CPU spends ½ of execution time in stalls (waiting for data to be

loaded from RAM to CPU cache) or cache misses

HANA

• Cache-aware memory organization – minimizes CPU stalls and cache misses

• Maximize spatial locality of data through the use of column stores


Column Store Advantages

• Higher data compression rates when redundancy is high

Ex. run-length encoding, cluster coding, dictionary coding

• High performance for column operations

Ex. searching/aggregations implemented as loops over arrays stored contiguously

• Elimination of additional indexes

Storing in columns is like getting a built-in index for each column

• Parallelization

Column store is already vertically partitioned, so operations on diff columns can be easily

processed in parallel

• Elimination of materialized aggregates

In-memory column stores allow calculating aggregates on-the-fly, which simplifies data

model, app logic, and concurrency


Column Vs Row Store

Use column store when:

Use row store when:

• Calculations are typically executed on single or a

few columns only.

• The table is searched based on values of a few

columns.

• The table has a large number of columns.

• The table has a large number of rows and

columnar operations are required (aggregate, scan,

etc.).

• High compression rates can be achieved because

the majority of the columns contain only few distinct

values (compared to number of rows).

• The application needs to only process a single

record at one time (many selects and/or updates of

single records).

• The application typically needs to access a

complete record (or row).

• The columns contain mainly distinct values so that

the compression rate would be low.

• Neither aggregations nor fast searching are

required.

• The table has a small number of rows (e. g.

configuration tables).

Deciding whether to use column or row stores depends on the situation…


HANA Architecture

Developed mainly in C++ and designed to run on Linux servers


HANA Architecture

Index server

• Contains data stores and engines for processing data

Preprocessor server

• Used by index server to analyze text data

Name server

• Contains info about topology of HANA system

• In distributed setup, knows where components are running and what data is on which

server

Statistics server

• Collects info about status, performance, and resource usage from all components

XS engine

• Maps persistence model in DB to a consumption model exposed to clients via HTTP


Distributed HANA

HANA setups are usually distributed over multiple hosts for scalability and availability

HANA system is identified by a system id (SID)

HANA instances are identified by instance ids (IID)

XS engine and statistics server only exist once per system


Index Server Architecture



Authentication

• Invoked when new connection established

• Users can be authenticated by HANA or an external provider

Authorization Manager

• Invoked by other DB components to check whether user has required privileges to execute

requested operations

Transaction Manager

• New sessions are assigned to a new transaction

• In HANA, each SQL statement is processed in the context of a transaction

• The transaction manager coordinates and tracks transactions

Request Processing and Execution Control

• Analyzes, executes, and forwards client requests

Planning engine

• Allows financial planning apps to execute basic planning ops in the DB layer

• Ex. create new version of dataset as a copy of existing one while applying filters and

transformations



Calc Engine

• Implements features like SQLScript, MDX and planning operations

Metadata Manager

• Manages metadata

• Ex. definitions of tables, columns, views, indexes, SQLScript functions, object stores

Relational Stores

• Subsystems of HANA DB which include in-memory storage and components that manage that storage

Persistence Layer

• Responsible for durability and atomicity of transactions (ex. DB restore on restart, log storage)

Repository

• Provides management and persistence of app specific objects

Ex. modeled views, stored procedures, web content exposed via XS engine

• Provides namespaces, versioning, and content import/export support

• Objects are stored with associated metadata (ex. type and version)


HANA Studio

Both the development environment and admin tool for HANA

• Dev: create modeled views or stored procedures

• Admin: start/stop services, manage system

• Developed in Java and based on Eclipse platform

Directory structure when connected to HANA

Catalog

• HANA‘s data dictionary

• Contains all data structures, tables, and data

Contents

• Design-time repository

• Hold all models created with modeler

Note:

• “Contents“ are physically stored in DB tables under “Catalog“

• Contents node just provides a different view on the same

physical data


Modeled Views

• Created using modeling tools in HANA studio

• Design-time artifacts stored in the repository

• For runtime execution, column store views are generated from modeled views

Advantages:

• Additional metadata can be provided (ex. attribute descriptions)

• Easier to use modeling tools compared to writing complex DDL

• Can be versioned in repository

• Easily consumable by customers

Modeled Object (design-time) Database View (run-time)

Attribute View Column view of type JOIN

Analytic View Column view of type OLAP or column view of type

CALCULATION on top of OLAP view

Calculation View Column view of type CALCULATION


Attribute Views

Define joins between tables and/or projections on a table

• Ex. joining multiple tables together to create a single dimension table when using star schemas; this can

be joined to a fact table via analytic view to give meaning to the data

Join employees

to org units, then

join result to

sales transaction

via analytic view


Analytic Views

• An OLAP cube (n-dimensional array of data)

• Set of physical tables interconnected in a star

schema (dimension tables around a fact table)

• Fact table provides info the user is actually

interested in (aka metrics, measures, or key figures)

• Dimension tables provide categories (classes, or

attributes) by which facts can be grouped and are

modeled as attribute views to enable reuse in

multiple analytic views

• Fact table provides content of OLAP cube

ie. “what” to group

• Dimension tables specify cube’s dimensions

ie. “by what” to group

• Cardinality of fact to dimension is N:1


Calculation Views

• Used to provide composites of other views (join or union of multiple data flows) or invoke built-in or

generic SQL functions

• Used the same way as analytic views

• Able to perform complex calculations

Graphical

• Modeled using graphical modeling features of HANA modeler

Scripted

• Created as sequence of SQL statements


Application Programming Options

Modeled Views (Calculation, Analytic, Attribute) & SQL Views

• Views are for used for defining application specific data models, for defining data flows and to structure

and reuse queries.

• Modeled analytic views and attribute views can be created with HANA studio without programming.

• Complex data flows can be modeled as calculation views using SQLScript or a visual modeler.

SQLScript

• The rich stored procedure language of the HANA.

• Procedures may contain SQL statements and call other procedures.

• Is used to write procedural orchestration logic and to define complex data flows.

R

• HANA also supports stored procedures written in “R” for statistical computing.


Application Programming Options

L

• Dedicated imperative programming language for implementing stored procedures and functions (known

as scalar functions) that can be used in an SQL query.

• Stored procedures in L are typically used to implement operators that are called from SQLScript

procedures.

• L code is compiled to native code, and thus very efficient.

C++

• Performance critical code can be developed in C++ as part of an application function library.

• Using C++ is restricted to specialists who cooperate closely with HANA developers.

FOX

• The planning engine supports planning operations written in the FOX formula language.


XS Engine Introduction

• Provides access to HANA DB by

transforming persistence model stored

in the DB into consumption model for

clients exposed via HTTP

• Uses SAP ICM for HTTP server

• Hosts system services that are part of

the HANA DB (ex. Search service, built-

in web server that provides access to

static content in the repository)

• Optional component of HANA

• Does not store data


Application Specific Code: Where?

Both the index server and XS engine can store application specific code. So what’s the difference?

Index server

• All data-intensive calculations that need to be done close to data in index server (using

SQLScript, modeled views, L procedures, C++).

XS engine

• Just provides consumption model for client apps (non-performance-critical tasks, invokes views

and stored procedures that are executed in the index server).


Index Server Vs XS Engine

XS engine is similar to

an index server without

data stores and with

an XS layer on top

When XS layer needs

to access data:

1. Opens a local DB

connection and

sends an SQL

statement to the

local SQL proc.

2. SQL proc

determines

location of data

and delegates

execution to index

server that has the

data.


Repository

• Repository tables and content reside on the index server like all other tables

• Used for storing and versioning source code of apps for the XS engine

• Provides lifecycle management

• Repo objects uniquely identified by a combination of its package name, object name, and object type

Advantages:

• Save / edit intermediate development results without affecting runtime objects

• Developers work with abstract repo objects and don’t need to worry about corresponding representation

in the HANA DB (ex. Writing DDL statements)

• Unified local dependency and version management (no need for external system, and helps in delivering

content to customers and among different HANA instances)

• Allows multiple users to work on multiple inactive versions of the same object in parallel


Parallel Development in Repository

• Repository uses lock-free approach

• Others must merge with the current active version before activation

• Repository checks inactive “based on” version equals current active version before allowing activation


XS Engine Architecture


XS Engine Architecture

1. Request handler uses URL mapper to determine the component that processes the request.

If there is a declarative consumption model in the repo for the requested resource, the OData

service is invoked. Otherwise, URL mapper determines which app in which container is

responsible for executing the request.

2. Request handler invokes the session manager (which also authenticates session)

3. Request handler checks with authorization manager to see if user is authorized to execute the

request

4. Request handler forwards request to OData service or responsible app container (app

container loads the app code from the repo)

5. Container triggers execution of the app code (app gets request object as input, processes

request and populates response object with output)


XS Engine Application Development

• Guideline for applications is to expose a consumption model via REST services based on OData

• The XS engine currently contains an OData service that supports a declarative mapping between

the persistence and consumption models

• When additional logic is required, application specific code needs to be written for processing client

requests

• The XS layer contains multiple application containers (executing environments)


XS Engine Application Containers

Currently, there are two major application containers:

C++

• Not used for general app development (special purposes only)

• Apps have same lifecycle as HANA DB and must be part of DB build

• Currently used for system apps

• Ex. web server for access to web resources stored in the repo

JS

• Based on Mozilla JS engine

• Currently used for pilot projects

• Due to lack of abstraction and encapsulation mechanisms, not recommended for complex

applications

??? (currently codenamed SLang)

• Additional app container is planned for the future that will have the benefits of JS, but also handle

existing JS shortcomings


XS Engine Application Development

Need a way to access repository in XS engine

• Regi is the currently recommended repo access tool

• Used to load and activate repo objects which includes both DB objects (ex. Tables) and app resources

(ex. xs*, html)

• Helps to track object state within the repo

• Provides workspaces on local dev machines for XS application development


Regi Workspace Structure

Below is an example of what a regi workspace looks like on your computer

packageTopLevel/

…

packageBottomLevel/

db/ Persistence

*.schema

*_TBL.sequence

*_TBL.table

logic/ Backend logic

*.xsjs

*.xsjslib

odata/ OData

*.xsodata

*.view

ui/ UI related

*.png

*.css

index.html

*.xsa

*.xscfgm

*.xscfgd

*_ext.xscfgd


XS Engine Application Files

Based on the workspace example:

Root

*.xsa

• Application configuration (JSON)

*.xscfgm

• Configuration model

• Describes config variable types

• Like a template

*.xscfgd

• Configuration data

• Specifies config values

*_ext.xscfgd

• Configuration data

• Overrides / extends configuration from base *.xscfgd file

index.html

• Main file for application

Persistence

*.schema

• Defines database schema

*_TBL.sequence

• Defines a sequence (JSON)

*_TBL.table

• Defines a table (XML)


XS Engine Application Files

Logic

*.xsjs

• XS JS

*.xsjslib

• XS JS library

• Contains functions that can be called from XS JS files

• Use ‘repo’ object of the XS JS API to include the library:

repo.import(<package name>, <object name>);

ex. repo.import(“sap.app.logic”, “config”);

• Call function from library with:

$.<package name>.<library>.<function>(<parameters>);

ex. val = $.sap.app.logic.config.getConfig(configParam);

OData

*.view

• Defines a view (JSON)

• Necessary if you wish to get data from multiple tables

*.xsodata

• Defines OData table (XML)


Inactive Vs Active Objects

Inactive Objects

• When committed to the repo, only visible to user who created it

• Can have multiple inactive versions of the same object in the same repo (differentiated by workspace)

• Using an attribute view as an example, contains description of the attribute view (ex. which tables to join)

Active Objects (may or may not include runtime objects)

• When an inactive object is activated, it becomes visible to all repo users

• If applicable, corresponding runtime objects are created using SQL DDL

• Using an attribute view as an example, this would be a column view of type “join view” (CREATE

COLUMN VIEW…)

• Other repository objects (ex. web content) do not have runtime objects created during activation


Deploying an XS Engine Application

1. regi track <package name>

Add package to set of tracked packages of the workspace

2. regi commit

Commit local changes to the repository (specify a list of files, or commit all tracked packages in your

workspace)

3. regi activate

Activates inactive versions of objects in the repository (specify a list of objects, or activate all inactive

objects in your workspace)


Resources

HANA Blue Book

https://portal.wdf.sap.corp/irj/go/km/docs/corporate_portal/WS%20PTG/Product%20Architecture/Knowl

edge%20Transfer/Bluebook/The%20SAP%20HANA%20Database.pdf

Views

https://tdwiki.pgdev.sap.corp/download/attachments/100634778/Analytic_Views_What_are_they_and_h

ow_do_we_use_them.pptx?version=1&modificationDate=1333127736180

XS Wiki

http://trexweb.wdf.sap.corp:1080/wiki/index.php/XS

HANA Studio Installation Files

file://vantipsoftware.pgdev.sap.corp/software/SAP/HANA/

HANA (includes XSE) Installation Files and Instructions

file://trextest/trex_test/xsengine

http://trexweb.wdf.sap.corp:1080/wiki/index.php/XS_Installation_Windows

HANA XSE Sample Application Tutorial

http://trexweb.wdf.sap.corp:1080/wiki/index.php/Sample_Application_Development_Drop8

https://portal.wdf.sap.corp/irj/go/km/docs/corporate_portal/WS PTG/Product Architecture/Knowledge Transfer/Bluebook/The SAP HANA Database.pdf



https://tdwiki.pgdev.sap.corp/download/attachments/100634778/Analytic_Views_What_are_they_and_how_do_we_use_them.pptx?version=1&modificationDate=1333127736180





//vantipsoftware.pgdev.sap.corp/software/SAP/HANA/

//vantipsoftware.pgdev.sap.corp/software/SAP/HANA/

//trextest/trex_test/xsengine

//trextest/trex_test/xsengine



http://trexweb.wdf.sap.corp:1080/wiki/index.php/Sample_Application_Development_Drop8


© 2013 SAP AG. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG.

The information contained herein may be changed without prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

National product specifications may vary.

These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or

warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group

products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing

herein should be construed as constituting an additional warranty.

SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in

Germany and other countries.

Please see http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark for additional trademark information and notices.

http://www.sap.com/corporate-en/legal/copyright/index.epx



HANA & XS Engine Overview

Documents

Transcript of HANA & XS Engine Overview