ActiveWarehouse/ETL - BI & DW for Ruby/Rails
-
Upload
paul-gallagher -
Category
Technology
-
view
8.676 -
download
1
description
Transcript of ActiveWarehouse/ETL - BI & DW for Ruby/Rails
![Page 1: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/1.jpg)
NB: This presentation was delivered at the Singapore Ruby Brigade meetup 6-Jan-2010 (at hackerspace.sg)
![Page 2: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/2.jpg)
BI & DW for Ruby/Rails
“!???”
![Page 3: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/3.jpg)
Why should we care about this enterprisey stuff?
Have you heard a client ask for..– A “dashboard”?– Management reports?– Operational statistics?
..in addition to the actual site?
![Page 4: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/4.jpg)
Or maybe you want to pitch for the dashboard/BI projects
themselves?
..using your rails skills of course
BIBusiness Intelligence
CPMCorporate Performance Mgmt
BPMBusiness Performance Mgmt
B&PBudgeting and Planning
EPMEnterprise Performance Mgmt
DashboardEnterprise Dashboards
![Page 5: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/5.jpg)
BI Basics
No, BI is not (always) an oxymoron
![Page 6: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/6.jpg)
Infrastructure & Systems
BI = Business Feedback & Control Systems
Keeping the doors open
Uptime on the
servers; alerts
![Page 7: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/7.jpg)
Infrastructure & Systems
OperationalManagement
BI = Business Feedback & Control Systems
Keeping the doors open
Optimising in the short termintra-day
Focus on systems in isolation
Need extra call centre
staff on shift?
Daily sales
numbers?
![Page 8: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/8.jpg)
Infrastructure & Systems
OperationalManagement
ExecutiveManagement
BI = Business Feedback & Control Systems
Keeping the doors open
Optimising in the short termintra-day
Focus on systems in isolation
Strategic performancemonthly, quarterly, yearly
Across all systems
Profitability by
product
Utilisation and sales
performance
![Page 9: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/9.jpg)
Infrastructure & Systems
OperationalManagement
ExecutiveManagement
Traditional Rails perspective..
e.g. NewRelic
Custom AR reports
Someone else’s problem (opportunity)
![Page 10: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/10.jpg)
Someone Else’s Problem..
Your Rails Storefront App
Fulfillment(maybe a third party)
To report on sales
fulfillment..
AR/AP/GL
To report on revenue and profitability..
To report on sales revenue,
actuals and forecast..
And don’t forgetall those other systems..
CRM MRP
FA
![Page 11: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/11.jpg)
Who is “Someone Else”?
The gigaohm network: “5 Free Business Intelligence Crunchers for Your 2010 Arsenal”
![Page 12: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/12.jpg)
![Page 13: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/13.jpg)
ODS
ETLYour Rails App
Other TransactionalSystems
Data Sources
DBoR, relational reporting
BI & DW
A copy of transaction data specifically structured for
query and analysis
Extract – Transform – Load
Or, Extract – Load – Transform
Or, Transform – Extract – Load
(depending on the technology)
![Page 14: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/14.jpg)
“cubes”
Sales = $22 Customer ID Product ID Date ID …
Customer dimension Date dimension
Product dimension
Fact categorisation“Fact”
![Page 15: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/15.jpg)
MOLAP, ROLAP, HOLAP
MOLAP: proprietary format to optimize for analytical queries
ROLAP: use relational database to mimic multi-dimensionality
HOLAP: hybrid. Drive analytics from MOLAP, drill down to relational
Star schema Snowflake
![Page 16: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/16.jpg)
Why?? What’s wrong with..
select a.name, sum(b.amount) from products ajoin order_items b on a.id =b.product_id group byproduct_id
Product.sum(:amount, :include=> :orders, :group =>‘product_id’)
•Every question needs it’s own query
•Can’t predict all the questions in advance
•Un-scalable grunt work
![Page 17: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/17.jpg)
ActiveWarehouse ActiveWarehouse-ETL
![Page 18: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/18.jpg)
ActiveWarehouse
Rails plugin by Anthony EdenROLAP solution based on ActiveRecordFeatures
– Generators for Facts, Dimensions, Cubes and Bridges
– Supports calculated fields– View helpers for reports with drill down
![Page 19: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/19.jpg)
ActiveWarehouse-ETL
Rails gem/plugin by Anthony EdenDSL for extract – transform – loadSource/sink: file, db, xml, .. (extensible)Features
– Pre/post processors– Transformations
![Page 20: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/20.jpg)
![Page 21: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/21.jpg)
The Cupcakes Store
Use Activewarehouse-etl to load seed data from csv to app db (mysql)
1
The Cupcakes BI Dashboard
2Use Activewarehouse-etl to load dimension and fact data to the warehouse(mysql to mysql)
3Use Activewarehouse to build a simple analytical “dashboard” and reporting tool
Follow the documentation at http://github.com/tardate/cupcakesinc to see how this works (and try it yourself)
![Page 22: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/22.jpg)
Product listing at Cupcakes Inc..
![Page 23: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/23.jpg)
Customer listing at Cupcakes Inc..
![Page 24: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/24.jpg)
Order listing at Cupcakes Inc..
![Page 25: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/25.jpg)
Order detail at Cupcakes Inc..
![Page 26: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/26.jpg)
Sales By Product AW Report
![Page 27: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/27.jpg)
Sales By Product (drill to 2009)
![Page 28: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/28.jpg)
Reasons to be Cheerful..
![Page 29: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/29.jpg)
Language
ETL processing, cube rules etc typically use custom languages (often archaic and limited)
BISuites
It’s … ruby!
![Page 30: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/30.jpg)
UI Customisation and Presentation Integration
Web delivery typically very constrained.
Often rely on strong integration with office software (Excel). Leads to “custom application development in Excel” syndrome.
BISuites
It’s … ActionPack!
Google maps mashups, social graph links. .. you get full UI control, as long as you have the development budget.
![Page 31: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/31.jpg)
Speed of development
Basic deployments can be very fast.
But UI inflexibility can lead to either lots of time wasted trying to shoe-horn, or need to “reset customer expectations”
BISuites
It’s … Ruby & Rails. Say no more ;-)
![Page 32: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/32.jpg)
TCO
Top-tier suites can come with a hefty $ tag. And prices are going up..
But some analysts are predicting 2010 to be the year BI gets FLOSS momentum (see gigaohm review of 5 well established alternatives)
BISuites
It’s … Ruby & Rails. Say no more ;-)
Trade-in software license costs for more development.
![Page 33: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/33.jpg)
Caveats..
![Page 34: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/34.jpg)
Native MOLAP
Generally good support for database MOLAP features.
Can be platform specific though – e.g. Microsoft MDX, SQL Server Analytical Services
BISuites
A gap. No real support currently available.
ActiveWarehouse uses relational model to “fake” MOLAP (ROLAP)
![Page 35: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/35.jpg)
Performance
Generally, all established analytical engines (and backing databases) have great performance track record. Huge scalability (millions of rows)
BISuites
Unproven. ActiveWarehouse/ETL does not have many (public) proof points.
Given that it is tied to AR performance, expect scalability could be an issue.
![Page 36: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/36.jpg)
Take-aways~ActiveWarehouse
It’s an impressive codebase. When you get it working, it works well.. but
– Virtually no documentation!– No contemporary examples– Not under very active development– A “textbook” data warehouse implementation.
May or may not be exactly what you want..
Remember: – data is batched. Not realtime.– Rails 2.x : install the plugin (gem is 1.x)
3
![Page 37: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/37.jpg)
Take-aways~ ActiveWarehouse-ETL
Neat tool. In addition to feeding AW:– Generate and load seed/test data– Move data between systems
But again,– Poor documentation– When it fails, can do so silently (makes
sure filename paths are delimited correctly for your platform!)
2
![Page 38: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/38.jpg)
Take-aways~ BI on Rails Solutions
Plain AR– just avoid the rabbit hole
AR + ETL– get all the data you need in one place
AW+ETL– traditional ROLAP, make Rails the focus of the BI effort
Go the BI suite route– When you need to adapt to many transactional systems
at scale, and customer has the $$ – (Rails remains just for transactional apps)
Or… (discussion point;-)
1
![Page 39: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/39.jpg)
Thank you!
Questions?
0
![Page 40: ActiveWarehouse/ETL - BI & DW for Ruby/Rails](https://reader035.fdocuments.us/reader035/viewer/2022062418/554f9f17b4c90586258b4865/html5/thumbnails/40.jpg)
Some References
ActiveWarehouse: http://github.com/aeden/activewarehouse ActiveWarehouse-ETL: http://github.com/aeden/activewarehouse-etl Cupcakes Inc sample site(s): http://github.com/tardate/cupcakesinc Singapore Ruby Brigade (SRB): http://groups.google.com/group/singapore-rb