Itaipu A Business Activity Monitoring system designed for business users Prof. Jacob Slonim June 27,...
-
Upload
claud-nicholson -
Category
Documents
-
view
213 -
download
0
Transcript of Itaipu A Business Activity Monitoring system designed for business users Prof. Jacob Slonim June 27,...
ItaipuA Business Activity Monitoring system designed for business users
Prof. Jacob Slonim June 27, 2009
Student: Azza Abouzied, MCSc
Motivation Related work Itaipu Usability Future workStream Cubes
Largest hydroelectric dam in the world, provides 20% of Brazil’s and 94% of Paraguay’s power
Outline
• Motivation
• Related Work
• Research Questions
• Our vision of BAM: Itaipu
• Multi-dimensional analysis
using Stream Cubes
• Usability
• Future Work
• Provide just-in-time analysis of business events– Business Intelligence (BI)
• Business Objects– Initiated research– Different approaches
Motivation Related work Itaipu Usability Future workStream Cubes
Research Goal
Motivation Related work Itaipu Usability Future workStream Cubes
• Enterprise’s Global Responsiveness and Agility
• Wal-Mart sales employees– Oil prices increase– Weather predictions– Canadian tire discounts garden products– Will we make the quarter?
Motivation
Motivation Related work Itaipu Usability Future workStream Cubes
Business Activity Monitoring (BAM)
• Continuous queries over data streams– Continuous query
• has implications on multiple query optimization– Infinite, unbounded, data streams
• Single-pass processing, processing bounds, or approximation techniques required
• Ad-hoc queries
• Dynamic environment
• Target business end-users NOT technical analysts
• Complex Event Processing Systems– Cayuga– Origin: Business Process Management– Business Objects Approach
• Data Stream Management Systems– STREAM– Aurora, Borealis– TelegraphCQ– Origin: Monitoring sensor networks
Motivation Related work Itaipu Usability Future workStream Cubes
Related Work
Motivation Related work Itaipu Usability Future workStream Cubes
Research Questions1. Could we reuse DSMS for a BAM system?
2. Could we provide multi-dimensional analysis of data streams?
3. Could we simplify and expedite query write-up for business users?
Motivation Related work Itaipu Usability Future workStream Cubes
Largest hydroelectric dam in the world, provides 20% of Brazil’s and 94% of Paraguay’s power
Itaipu• Spring
– Data Source
• Dam– Query processing
unit
• Delta– Results display and
storage
Motivation Related work Itaipu Usability Future workStream Cubes
Itaipu’s big picture
Motivation Related work Itaipu Usability Future workStream Cubes
• Brief design overview of Itaipu
• Two more questions left:
– Multidimensional data analysis of stream data
– Enhancing user-experience or usability
Recap & What’s Next?
city, sub_category
city, categorycountry, sub_category
country, category
country, allall, category
all, all
city, allall, sub-category
Cuboid
Base cuboid
Apex cuboid
Motivation Related work Itaipu Usability Future workStream Cubes
Cube: a multi-dimensional data model
• Dimensions– Region: (city < country
< all)– Product: (sub-category
< category < all)
• Measures– Sum, count,
average
• SizeProduct
dimensionRegion dimension
Motivation Related work Itaipu Usability Future workStream Cubes
Stream Cubes
• Han et. al introduced stream cubes
• Minimally-interesting (m-layer) and observation layers (o-layer)
• Partial Materialization– Popular drilling path
Motivation Related work Itaipu Usability Future workStream Cubes
Conceptual representation of a tilted time frame
Stream Cubes
• Tilted Time frame– Time dimension– Distant aggregates
provided at larger granularities in comparison to more recent ones
Motivation Related work Itaipu Usability Future workStream Cubes
Difference in perspective
• Popular drilling path– Users interested in o-layer and occasionally drill down– Path could be selected by an expert– Users drilling path is not likely to change
• Dynamic materialization path– User requests are distributed at different abstraction
layers– User requests are likely to change or evolve over the
course of the cube’s lifetime in the system
city, sub_category
city, categorycountry, sub_category
country, category
country, allall, category
all, all
city, allall, sub-category
Motivation Related work Itaipu Usability Future workStream Cubes
Drivers for change• External factors
– Oil prices increase– Export and oil-based products
affected– Shift towards viewing more
product information, less regional detail
• Competition• Seasonal• Market or economic
Motivation Related work Itaipu Usability Future workStream Cubes
Dynamic Materialization Path
• Simple and efficient approach– Linear cost-effectiveness function– Dynamic programming approach
• Cost-effectiveness function– Memory cost– Popularity effect– Update potential
• Adapts to different operating environments– Weighted cost-effectiveness =- α(Memory cost) + β(Popularity) + γ(update potential)
Motivation Related work Itaipu Usability Future workStream Cubes
Update potential?
Motivation Related work Itaipu Usability Future workStream Cubes
α = 0.5,
β = 1.0 ,
γ = 0.75
Simulated example
Motivation Related work Itaipu Usability Future workStream Cubes
city, sub_category
city, categorycountry, sub_category
city, allcountry, category
country, all
all, all
all, category
all, sub_category
Resetting Path
Motivation Related work Itaipu Usability Future workStream Cubes
Resetting paths on demand
Motivation Related work Itaipu Usability Future workStream Cubes
Benefit of dynamic path materialization?
• Flexible– Follows user requests
• Low reliance on IT staff– Suitable for just-in-time operation
• How does it affect data stream processing?– Queue Size measure
• How does it affect memory consumption– Memory usage
Motivation Related work Itaipu Usability Future workStream Cubes
Data rate ≈ 12,000
Experimental Setup
Motivation Related work Itaipu Usability Future workStream Cubes
Experimental Setup
Motivation Related work Itaipu Usability Future workStream Cubes
Results – Queue size
Motivation Related work Itaipu Usability Future workStream Cubes
Results – Memory consumption
Motivation Related work Itaipu Usability Future workStream Cubes
So, is it better?
• Conclusion– Situation dependent: more flexible solution– Not suitable when fixed memory guarantees are
required
• Are their better dynamic path materialization algorithms?– Probably– Our aim: simple, efficient, something that works!– Future improvements?
Motivation Related work Itaipu Usability Future workStream Cubes
Recap & What’s Next?
• Brief design overview of Itaipu
• Optimization work with stream cubes
• Last question:– Enhancing user-experience or usability for Itaipu’s
business end-users
Motivation Related work Itaipu Usability Future workStream Cubes
BAM user experience
• Bottleneck to the adoption of BAM– Operational decision making requires query
enabling technology
– The entire organization utilizes BAM: training effort needs to be minimal
Motivation Related work Itaipu Usability Future workStream Cubes
A usability framework
Jarke’s & Vassiliou’s model:• Training Effort
• Repeated Efforts– Thinking
• Self-construction• Assisted-construction
– Input– Correction
Motivation Related work Itaipu Usability Future workStream Cubes
Lessons Learned from database usability
• The simpler the task, the less query formulation effort is required.– Corollary: Limiting the functionality of a query
interface, makes it easier to use– Query by example– Query by forms
• Collaboration reduces the training effort– C-TORI– SearchTogether
Motivation Related work Itaipu Usability Future workStream Cubes
Redrawn from a table in Jarke et Vassiliou, 1985
BAM’s user profile
Motivation Related work Itaipu Usability Future workStream Cubes
Our usability solution
• Query Frames– Created by data analysts– Simplify query formulation to completing a query outline– Transformed easily to a form interface
• Query search– Each query or query frame is associated with textual
descriptions– Users and analysts could share queries and query frames– Passive collaboration– Keyword-based Google-like search interface
Motivation Related work Itaipu Usability Future workStream Cubes
Query Frames
Motivation Related work Itaipu Usability Future workStream Cubes
Dimension Query Frame
Motivation Related work Itaipu Usability Future workStream Cubes
Dimension Query Frame
Motivation Related work Itaipu Usability Future workStream Cubes
Itaipu’s Search Interface
Motivation Related work Itaipu Usability Future workStream Cubes
Itaipu’s Search Interface
Motivation Related work Itaipu Usability Future workStream Cubes
Analytical Evaluation
Goal Query Frames Search
Training Thinking
Self construction Assisted construction
Input Correction
Motivation Related work Itaipu Usability Future workStream Cubes
Hypothesis Result
A DSMS could be used as the core of a BAM system Dynamic path materialization is better suited for the data stream model Query entry could be simplified with the help of query frames and query sharing ≈
Final Recap
Motivation Related work Itaipu Usability Future workStream Cubes
• Further studies to validate some our hypotheses
• A vision for BI
What’s in the future?
Motivation Related work Itaipu Usability Future workStream Cubes
Conclusion
• Business Objects was convinced with our solution:– Itaipu’s architecture– Stream Cube extension– Query frame and query search
1 0
Thank for the support of:
NSERC, Business Objects,
Business Intelligent
Network
Questions ?
Spring
Motivation Related work Itaipu Usability Future workStream Cubes
• Framework, based on data sources,– define
• Properties (allows drop)
• Data rate• Push/pull
– create:• Data Transformer• Data Streamer
• Data specification
Dam: Execution Engine
Motivation Related work Itaipu Usability Future workStream Cubes
• Execution Engine– Variant of pipe &
filter architecture
– Similar to Fjords model
– Works like an Operating system
• Thread Manager• Queue Manager• Operator Builder
Execution Engine: Queue Manager
Motivation Related work Itaipu Usability Future workStream Cubes
Execution Engine: Queue Manager
Motivation Related work Itaipu Usability Future workStream Cubes
Execution Engine: Thread Manager
Motivation Related work Itaipu Usability Future workStream Cubes
Execution Engine: Operator Builder
Motivation Related work Itaipu Usability Future workStream Cubes
• XML multiple-query plan– Decouple query planning/optimizing from execution engine
• Java to XML Binding (JAXB)– Builds a content object model for each XML element
representing an operator
• Framework for building operators– Hierarchy of builders (Operator -> Aggregators -> SUM)– Update existing network of operators
Dam: Query Processing Unit
Motivation Related work Itaipu Usability Future workStream Cubes
• Query Language– SQL-like with windowing constructs– Industry standard
• Current implementation– Extremely naïve– Open source tools
Delta
Motivation Related work Itaipu Usability Future workStream Cubes
• End-user interface– Displaying and storing results– Searching for queries in the query catalog or inserting
new queries using SQL or modifying query frames
• Communicates with execution engine – Delta is thick client, execution engine is the server– XML messages over a TCP channel
• Memory cost
Example c1 = country, c2 = product category, |c1| = 5,
|c2| = 10, then |(c1, c2)| = 5 * 10 = 50
A good cuboid
Motivation Related work Itaipu Usability Future workStream Cubes
A good cuboid
Motivation Related work Itaipu Usability Future workStream Cubes
• Popularity
• Update Potential
Cost-effectiveness function
Motivation Related work Itaipu Usability Future workStream Cubes
• Cuboid
• Path