In Memory or Live Data

8/2/2019 In Memory or Live Data

1/6

In-Memory or L ive Data:Which is Bet ter?

AUTHOR:Ellie Fields, Director Product Marketing,

Tableau Sof tware

DATE:July 2011


2/6

The short answer is: both. Companies today are using both to dealwith ever-larger volumes o data. For the business user, analyzinglarge data can be challenging simply because traditional tools areslow and cumbersome. Some companies are dealing with this byinvesting in ast, analytical databases that are distinct rom theirtransactional systems. Others are turning to in-memory analytics,which lets users extract set o data and take advantage o thecomputing power on their local machine to speed up analytics.

So which approach is better? There are times when you need towork within the com ort o your own PC without touching thedatabase. On the other hand, sometimes a live connection to adatabase is exactly what you need. The most important thing is notwhether you choose in-memory or live, but that you have the optionto choose.

Lets look at some scenarios where in-memory or live data might bepre erable.

In-Memory Data is Better

When your database is too slow or interactive analytics

Not all databases are as ast as wed like them to be. I youreworking with live data and its simply too slow or you to dointeractive, speed-o -though analysis, then you may want to bringyour data in memory on your local machine. The advantage oworking interactively with your data is that you can ollow your traino thought and explore new ideas without being constantly sloweddown waiting or queries.

When you need to take load o a transactional database

Your database may be ast, it may be slow. But i its the primaryworkhorse or your transactional systems, you may want to keep allnon-transactional load o it. That includes analytics. Analytical

queries can tax a transactional database and slow it down. So bringa set o that data in-memory to do ast analytics withoutcompromising the speed o critical business systems.

When you need to be o fine

Until the Internet comes to every part o the earth and sky, youreoccasionally going to be o ine. Get your work done even while notconnected by bringing data in-memory so you can work with it righton your local machine. Just dont orget your power cord orbatteryyoull still need that!

2011 Tableau So tware


3/6

A Live, Direct Data Connection is Better

When you have a ast database

Youve invested in ast data in rastructure. Why should you have tomove that data into another system to analyze it?

You shouldnt. Leverage your database by connecting directly to it.Avoid data silos and ensure a single point o truth by pointing youranalyses at a single, optimized database. You can give businesspeople the power to ask and answer questions o massive data sets

just by pointing to the source data. Its a simple, elegant and highlysecure approach.

When you need up-to-the minute data

I things are changing so ast that you need to see them in real-time,you need a live connection to your data. All your operationaldashboards can be hooked up directly to live data so you knowwhen your plant is acing overutilization or when youreexperiencing peak demand.

And the Best Choice? Both, o Course.

Even better is when you dont have to choose bet ween in-memoryand live connect. Instead o looking or a solution that supports oneor the other, look or one that supports choice. You should be able toswitch back and orth bet ween in-memory and live connection asneeded. Scenarios where this is use ul:

You want to use a sample o a massive data set to fnd trendsand build your analysis. You bring a 5% sample o the datain-memory, explore it and create a set o views you want toshare. Then you switch to a live connection so your reports areworking directly against all the data. Publish your views, andnow your colleagues can interact with your analysis and drill

down to the part o the data most relevant to their work.

Youre ying to New York and want to do some analysis on theplane. You bring your entire data set, several million rows, intoyour local PC memory and work with it o ine. When you get toNew York, you reconnect to the live data again. Youve doneyour analysis o ine and in-memory, but you are able to switchback to a live connection with a ew clicks.



4/6

Neither in-memory nor live connect is always the right answer. Iyoure orced to choose, youll lose something every time. So dontchooseor rather, choose as you go. Bring your data in-memory,then connect live. Or bring recent data in-memory and work o ine.Work the way that makes sense or you.

The Tableau Data Engine

Tableaus Data Engine provides the ability to do ad-hoc analysisin-memory o millions o rows o data in seconds. The Data Engineis a high-per orming analytics database on your PC. It has the speedbenefts o traditional in-memory solutions without the limitationso traditional solutions that your data must ft in memory. Theres nocustom scripting needed to use the Data Engine. And o course, youcan choose to use the Data Engine, or dontyou can alwaysconnect live to your data.

Data Engine to live connectionand back

The Data Engine is designed to directly integrate with Tableausexisting live connection technology, allowing users to toggle witha single click between a direct connection to a database andquerying an extract o that data loaded into the Data Engine (andback).



5/6

Architecture-aware design

The core Data Engine structure is a column-based representationusing compression that supports execution o queries withoutdecompression. Leveraging novel approaches rom computergraphics, algorithms were care ully designed to allow ull utilizationo modern processors. The Tableau Data Engine also has been builtto take advantage o all the di erent kinds o memory on a PC, soyou can avoid the common limitation that data sets must ft intoyour computers RAM memory. This means you can work withlarger data sets.

True ad-hoc queries

The Data Engine was designed with a query language and queryoptimizer designed to support typical on-the- y business analytics.When working with data at the speed o thought, it is common toneed to run complex queries such as very large multi-dimensionalflters or complex co-occurrence queries. Existing databasesgenerally per orm poorly on these types o queries, but the DataEngine processes them instantly.

Robin Bloor, Ph.D. and ounder o technology research frm theBloor Group, writes: Because Tableau can now do analytics so

swi tly and gives people the choice to connect directly to astdatabases or use Tableaus in-memory data engine, it has becomemuch more power ul in respect o data exploration and datadiscovery. This leads to analytical insights that would most likelyhave been missed be ore.

Flexible data model

One o the key di erences o the Tableau Data Engine compared toother in-memory solutions is that it can operate on the data directlyas its represented in the database on disc. So theres no requireddata modeling and no scripting that needs to be done to use theData Engine. One o the things thats so power ul about the DataEngine is you can defne, just as with any other relational database,new calculated columns or you might think o it as sort o ad hocdata modeling at anytime.

Instance load and connection time

The Data Engine is unique in that once your data is loaded into theData Engine, it has very ast start-up time. It only needs to read inthat portion o the data which your queries actually touch. Youmight have a lot o data in the database thats not relevant to aparticular analysis, but you are never going to wait or the TableauData Engine to read that data. 2011 Tableau So tware


6/6

The Customer Perspective

Nokia uses Tableau in-memory or marketing analysis. RaviBandaru, Nokias Product Manager or Advanced Data Visualizationsand Data, said, Its letting the analyst do more analysis himsel orhersel without IT coming between them and their data, he said.Using this kind o in-memory capability, I do see this being use ulin exploring more complex and largish data sets, which wereinaccessible be ore. 1

Playdom, a leading social gaming company owned by The WaltDisney Company, uses Tableau to directly query a ast analytical

Vertica database. Vice President o Analytics David Botkin describedhow they use Tableaus direct connect ability to analyze massivedata:

I a product manager is trying to understand some problem orsome dynamic in their game and using Tableau, they will typicallyrun against our Vertica data warehouse. That data can be anywhere

rom ew million to a hundred million rows. The product managerwill typically run a bunch o di erent cuts o the data using Tableauto look or patterns or to understand the behavior theyre a ter. Iwere querying a data set thats tens o millions to a ew hundredmillions o rows in Vertica using Tableau, we expect to get that databack in tens o seconds or less. So, its a very use ul tool or helping

people to iterate as they think, and react to answers that they haveseen to ask the next question.

Recommendations

Because o the variety o ways people need to work with data, werecommend that when choosing a business intelligence solution:

Ensure it provides the option to take data in-memory to speedup your analytics.

But dont limit yoursel to a solution that requires you to bringall data in-memory be ore you can analyze it.

Evaluate how easy it is to switch back and orth betweenin-memory and live connect: i you need to do custom scripting,it may take away most o the exibility you got rom yourchoice.

Examine care ully the data size limitations o your in-memorysolution. Choose an in-memory solution that lets you deal withvery large data sets.


1 Analytics at the Speed of Thought, Robin Bloor, Ph.D, July 2011.

Because Tableau can

now do analytics so

swiftly and gives people

the choice to connect

directly to fast databases

or use Tableaus

in-memory data engine, it

has become much more

powerful in respect of data exploration and data

discovery.This leads to

analytical insights that

would most likely have

been missed before.

- Robin Bloor, Ph.D and founder

of The Bloor Group

In Memory or Live Data

Documents

Transcript of In Memory or Live Data