Post on 01-Apr-2015
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
1 M
ay
12,
20
10©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
1
How (and Why) to Build a Data Warehouse 101
From Questions to Eight Steps to DW Heaven!
@aupward & #meshU
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
2
Declaration of Data Independence
When in the Course of an increasingly competitive global economy it becomes necessary for one data set to dissolve its connections to a constraining environment, the separate but inherently unequal station to which the Laws of Whose budget is larger prevails…. <snip>
We hold these truths to be self-evident, that all data is created equal, that they are endowed by their Creator with metadata that holds important context and critical unalienable rights, that among these are compliance, security and the right to drive business value…
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
3
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
4
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
5
And finally for the database query writers…
http://xkcd.com/327/
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
6
Who is Antony Upward• Funny after lunch?
• Management Consultant – Business Systems – 20+ years• Business Analyst, Project Manager / Program Director, Business Architect
– MIS @ Apple – GUI to Data Warehouse over WAN (DECNet) in 1990!– SAP @ Bell family of companies and elsewhere – including full SAP Data
Warehouse Implementation
• Academic… Teaching / Learning– Ryerson University School of Management
• Business Technology Management Program– Business Process Analysis and Design– IT Governance and the Role of the CIO
• Producing the people you need / want to hire– Understand Business AND Technology
New…Edward James Consulting… Sustainability Business Architect– Aligning People, Process and Technology to achieve Sustainable Results…
• Sustainable for People, Planet and Profits– Returning to University… Masters in Environmental Studies with Graduate
Diploma in Business and the Environment @ York / Schulich
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
7
Why?• Your world is…
– Complex– Dynamic– Data… coming at you like a fire hose
• How do you:– Make sense of your world?– Make informed decisions to:
• Lower costs, increase revenues, increase productivity?– Make plans… and know if you’re achieving them?
The Data Warehouse – a key tool
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
8
Your Talent Pool
Your Market
You’ve Launched…
Your Company
..and You’re Starting to see Your World is Complex…
Potential Customers
YourCustomers
YourSuppliers
Your CommunitiesCustomer – Supplier – Industry
YourBank
YourInvestors
YourAdvisors
(Accountant,Lawyer, etc.)
YourRegulator
Employees
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
9
…and you have systems…
Finance
Inventory/Warehouse
GL A/P
A/R
AssetsBilling
Projects Order Entry
Customer Care
Web Site ContentManagement
Purchasing
Internet / Cloud
Retail Web Site /
Portal
Wholesale& SupplierWeb Sites
…which you’ve started to interconnect…
Supplier
Bank
Community
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
10
…and you have transactions…Customer
Sales Orders
Purchase Orders
CustomerInvoices
SupplierInvoices
CustomerPayments
Payments toSuppliers(Cheques)
Proposals & Quotes forCustomer
Customer Quotes
ProposalsFrom
Suppliers
WebsiteActivity
(browsing, choosing,sharing, using…)
CustomerInquiries Praise& Complaints
Financial JournalEntries
Community(blog postings,
tweets/buzz, wall postings…)
Payroll
MarketingCampaigns
(Qualified Prospects)
…more and more all the time…
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
11
… and you have started to realize your data has structure …
• Transactional Data– Data related to specific business events
• Master data– Data related to the objects involved in your business, which change over
time – but remain constant over many transactions
• Meta data– Data about your transactional and master data
… perhaps realizing that all is not right…
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
12
…but most of all you have questions• How many customers do I have?
• When did the last buy from me?
• What did they buy?
• Where are my customers?
• What do my customers have in common
with each other?
• Which combinations of products do they
buy…so I can make recommendations?
• How did they behave on the website when
deciding what to buy?
• How are they using my application, product
or service?• Who is paying me on time / late?
• Who has upgraded and what led them to
decide to upgrade…so I can encourage
other customers to do the same?
• Which suppliers ship to me reliably (on-time, right quantities, no DOAs)?
• Which suppliers invoice me accurately?
…and despite all this data… …you don’t have ready answers!
• What do my customers tell me they like?
• What is being discussed about my company
in the community?• What are the trends in the complaints I
receive … how can I improve?
• Am I easy to do business with?
• Am I meeting my delivery promises to my customers? (no back-orders, on-time delivery, etc.)
• Am I spending the right amount on my inventory? (too much, not enough)
• How long does it take for me from getting an order to getting paid?
• Is every order profitable?• Did I make money in the last month, week,
day, hour?• How much did I spend on X?
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
13
… an aside… wondering why… if we have all the data why we don’t have information?
– The way we build systems* to help run our transactional business processes … rarely has anything to do with the information we need to manage, plan, change those same processes†
• Plus…– Data is very very slippery‡… it requires huge business
discipline to keep it all consistent over time… and soon as data is inconsistent it is hard to turn it into information
* Historically this was because we had no choice – it simply wasn’t possible to cost effectively buy the hardware or build the systems with the complexity to meet both needs simultaneously
† New technologies like in-memory databases will change this… but it will take 5-10 years before it is normal for transactional systems to also be able to provide management information
‡ Despite the claims of the semantic web people we are no closer to solving this problem! Don’t believe the hype
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
14
Management Process(OLAP)
OLAP = on line analytical processing (aka data warehouse)
The Big Picture…Where the data Warehouse Fits in Running Your Business…
Plan
Transaction(e.g. Take Order)
Transaction(e.g. pack & ship)
Transaction(e.g. invoice)
Act
Operational
Process
(OLTP)
Delivery of Organizations Value Proposition to Stakeholders
OLTP = on line transaction processing (aka ERP, CRM, etc.)
Analyze
Measure
Gain Understanding…
Answer Your Questions
Make Decisions…
Decide How to Action Them
Transactional Systems
1...n
Transactional Systems
1...n
Transactional Systems 1...n
THE data warehouseTHE data warehouse
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
15
So…You Need a Data Warehouse (DW)…
How Do You Get One? OR
Eight Steps to DW Heaven
* Like data warehouse technology sales people!
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
16
First…Recognize That Building a DW is Both Like and Unlike Building Other Systems…
Same…you need:• People… who will be using the
DW to be involved in designing, building and using it IF you want an ROI
• Process… a great project manager who has the right plan – one customized to your needs
• Technology… and great people who know the technology
• Don’t let anyone* tell you they know the answer and can “magically” give it to you tomorrow
Different…• The users of a DW are YOU… Your
leaders / managers!– Are you ready to get involved?– Are you ready to change – make
decisions using the information from the DW?
• You really don’t know what you don’t know about your data… get comfortable learning by iterating– You’ll spend less and get more
* e.g. DW technology vendor or consulting vendor sales people! There is a lot of hype out there. Be cautious.
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
17
Second… Do You Have the Skills You Need?
Enterprise Data Architect*(Data Modeller, Entity Relationship Diagrams)
* Technological owner of the definition of all data in the transactional and DW systems – not to be confused with the business owner
DataBase Knowledge(Oracle, SQL Server,
etc.)
DBA(Administrator)
Query Writer(SQL etc.)
DW Infrastructure Specialists
(Hardware, Operating System, etc.)
Your Management Team
Whose questions is the reason the DW is being Built!
Business Intelligence (BI)
Business Analyst
Understands the tech…
But can learn, understand and
talk to you YOUR business, YOUR
questions
DW Tool Experts
..In which ever tool you
decide touse†
† You’ll need more of this resource early on than later… so perhaps a good place to use contractors/consultants … at least initially and only hire later when you know the level of on-going need for this skill set
Plan to get the skills you need at the right time… 1a. BI Business Analyst, 1b. Enterprise Data Architect 2. Tool Expert
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
18
...Third… Decide on What To Do First…
1. Make a list of the most pressing questions you can’t answer today– Get your management team involved
2. Brainstorm what the answers might be… and what you will change if that answer turns out to be correct– Get everyone used to the idea that the DW is a tool which will become a
normal part of the process by which you will:• Make decisions , Make and prioritize plans for improving your
business• Drive change, Measure management team member success
3. Determine the potential benefits if you were to decide to implement the changes… Your Business Case– Which answers would drive the biggest benefits?
4. Pick the n questions whose answers would enable the biggest benefits
• 2 < N < 10– Drop the rest for now… you need to focus… DON’T get side tracked…
focus!
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
19
Fourth…Which Data Do You Need ?1. Mock-up the output from the DW which will answer each
question– Show it to everyone… listen to the feedback
• Remember why you are doing this… it is the ability to decide to change as a result of answering the question that matters not what the screen / report looks like!
2. How do you “calculate” each part of the answer?– Get everyone involved in figuring this out… – Decide which senior manager “owns” each calculation / measure
3. What data is required?– Identify both the master data (customer master, product master, etc.) and
the transactional data (sales orders, payments, etc.)– You do have an enterprise data architect and model don’t you?
• If not time to:– Hire that person– Build that model…and have your transactional systems people keep it up to date
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
20
Fourth (part Deux)… Where Is Your Data?
1. Where is the data?– Does it exist… ask the enterprise data architect… if not can you
substitute… – Start the feedback loop to future versions of your transactional systems
2. Is that data clean (consistent), what anomalies exist in that data?– Normal to find data isn’t clean… expect work arounds– Ensure everyone knows how these imperfections will impact the
accuracy of the answer– Start the feedback loop to future versions of your transactional systems
• Who in the business is responsible for cleansing the data and keeping clean
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
21
…Fourth (part Trois)… Get the Tech Ready
1. Set a budget for the tech– Use the benefits of answering the first group of questions to decide how
much to spend
2. Decide on the tools– Ensure they can grow with you– Get help deciding
3. Hire the DW tools expert(s)– The people who know the tool you’ve chosen– Consider contractors or consultants
4. Set the technology standards and design principles– Will help ensure reliability, flexibility, agility– Response time, refresh frequency,– Data Integrity, Data Security, Disaster Recovery, Business Continuance– Change Control (Development, Test, Production, Training)
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
22
The Data Warehouse
“Close Loop”
Analytic Tools
“Close Loop”
Analytic Tools
Make your decisions “real” by pushing process changes directly into operational / transactional systems. Can make changes in near real time!
The (Tech) Parts of a Data Warehouse
Your Transaction Systems(OLTP)
Your Transaction Systems(OLTP)
TransactionalDatabases
Clean Master
Data
Clean Master
Data
CleanTrans-
actional Data
CleanTrans-
actional Data
Dimension Tables
(Shared)
Dimension Tables
(Shared)
Extract, Transform and Load (ETL)‡
Extract, Transform and Load (ETL)‡
‡ Includes: matching, cleansing, versioning.Can be “pull” or “push” depending on volumes
Meta Data†Meta Data†
† Data about the data in the ODS and Cubes – to allow users to understand, and “self document” their queries
DW Control &
OperationsData
Includes: scheduling, reporting schedules, extract, transformation, refresh, DR rules etc.
Cube 1*Cube 1* Cub
e 2Cube 2
Cube nCube n
* Each cube (aka Data Mart) has the” fact” tables containg the transactional data transformed and the applicable dimension tables to answer groups of related questions
OperationalData Store (ODS)
aka Persistent Staging Area (PSA)
Technical tools – to be used by DW Tool Experts and BI Biz
Analysts
Read from & Write results to cubes
End User
Analytic Tools
End User
Analytic Tools
Read the cubes
End UserQuery & Reporting Tools
End UserQuery & Reporting Tools
Manual Data MaintenanceManual Data Maintenance
These need to be easy to
use… Management
will the users!
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
23
…Fifth… Build & Test a Prototype …Iteratively
1. Document the detailed design (BI Analyst and DW Tool Techs)
2. Take earlier mock-ups (step 4) and build a prototype– focus on getting to 80%– Expect the build to take 5-10x longer to answer first 2-3 questions than
questions 3-10• There is a lot of one time set-up…• Make sure your tech team know this is a prototype…
3. Test with users– Set expectations… people should be starting to get excited that its
becoming real… not upset because its not perfect!– Can your management now answer the questions they had and (more
important) can they make (and execute) the decisions based on the answers
4. Expect this step to take 2-12 weeks depending on complexity– Time box it at 12 weeks… reduce the scope
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
24
…Sixth… Use it… Make Decisions (Mistakes)… Learn… Improve
• Get the prototype so it can deliver answers– Expect regular manual effort
• Cleansing data, running ETL or reports, hand holding users if UI isn’t perfect
• Use the answers to make and execute decisions• Have a party… you have the start of a successful data
warehouse!• Learn
– After 2-12 weeks (depends on decision frequency) have a formal lessons learned
• Are you getting the benefits you expected?– Involve everyone – leaders, managers, BI Analyst, Tech team… you all
need to understand the challenges and what’s been learned– This is the difference between great high performing (BI/DW) teams
and mediocre ones… their ability to learn together • Plan the changes needed to have the DW reliably deliver the
answers
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
25
…Seventh… Productionize…The real go live…1. Formal training of users
– Prepare documentation– Make them confident in the use of the tools– Empower them with great meta data
2. Mechanising manual steps3. Scheduling of all the jobs
– make it run smoothly– No wee hours of the morning handholding!
4. Implement disaster recovery – If your DW is enabling operational decisions your DW is mission critical!
5. Party some more– You’ve been working hard at this for 3-8 months at this stage and can
now see real benefits happening!– Every day your decisions are being informed by your data via the DW
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
26
… Eighth… Now… Which Questions Didn’t You Answer?… Start Wave #2
• Build on your success…– Go back to Third step and repeat…but go faster!
• Remember incremental cost of answering more questions and adding more data to the warehouse is much lower than the first wave due to:– One time investments made – The experience you have
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
27
You Are On Your Way to a Sustainable Business
• Having all management decisions and strategic planning informed by a single, consistent set of information which your management team all align around is the basis for the long term health of your business
• The DW is just a tool, but one which when used with significant management / leadership discipline can make the difference between long term success and failure
• The DW is not a one-time project… it is a tool to enable a better way of running your business… forever!– Expect to want to evolve it as you and the DW tools become more and
more sophisticated
Ho
w (
and
Wh
y) to
Bui
ld a
Da
ta W
are
hous
eV
ers
ion
1.0
May
17
, 201
0©
Mes
hU
pre
par
ed
by E
dw
ard
Ja
me
s C
on
sulti
ng
28
Thank-you… download slides from http://www.EdwardJames.biz/documents