Traditional vs. Agile BI

INTRODUCTION

Since Business Intelligence emerged into mainstream awareness

in the 1990's, the imperative of delivering a "single version of

the truth" has been an extremely challenging vision to realize in

most organizations. Legacy BI has centered on the assumption

that more information yields better decisions, and other than

support for highly routine decisions made in mature, stable

environments, this model has largely resulted in failure. This is

mainly caused by three factors: 1) the difficulty and time

required to integrate all of an organization’s data before any

analysis can be done has resulted in extremely challenging

implementations; 2) the fact that legacy BI technology has been

optimized not for how decisions are made, but rather for solving

technical limitations (many of which have been removed or are

rapidly being removed); and 3) traditional BI platforms are

unable to adapt to system change, which is inevitable given that

we operate in a competitive environment where everything is

dynamic.

Traditional BI technologies have focused on solving data storage,

integration, processing, and presentation issues. With the goal of

decision support left unachieved, a new model of BI has

emerged called Agile BI, which is built on many opposing

assumptions like looser data integration, the utilization of less,

more targeted information to make decisions, and the reality of

continuous system change. Agile BI changes the focus from

data driven to decision driven.

MORE IS BETTER

Legacy BI is based on the assumption that by having access to

every piece of information about every aspect of a business

process, we can make better decisions. This so called “single

version of the truth” has lead to the ideal of the “Enterprise Data

Warehouse”, in which a complete, unified view of our entire

enterprise can be found. And by knowing everything, in context

of everything else, our decision making will be fool proof.

This is a very attractive idea that unfortunately just doesn’t work

in practice. Even if it is theoretically possible to construct a

unified, complete “single version of the truth,” it is likely the

competition will have outmaneuvered you long before you are

able to act upon it. And in competitive environment, the “truth”

changes as new markets open, as new competitors come onto

the field, and as market dynamics change the game.

Not only is it impractical for most organizations to build a

universal view, but current research in the decision sciences also

indicates that decision models that attempt to include all

available information actually don’t perform well in the real

world. Such models do a great job of “predicting” the data you

already have, but fail to work in new situations. And if you want

humans to participate in the decisions, understand them, and

take action, more information inputs and complexity leads to

poorer adoption as well as difficulty in judging when the model

might be failing. These factors have been echoed in poor BI

adoption rates and the spectacular failures of so-called “data

driven” organizations in the recent financial industry crises.

Agile BI focuses on the requirements of the decisions being

made, rather than on corralling all available data. Data may be

tightly integrated to support decisions, or it may be loosely

joined without the need for conformed dimensional models. As

the decision model changes, information that seemed critical

may fade in importance and new data source requirements will

emerge. To be Agile, BI must quickly integrate (and dis-

integrate) this information for the decision maker.

THE “TRUTH” CHANGES CONSTANTLY

The success of Agile Methodology in software development is

largely due to the fact that it accepts constant change as the

norm. Every principle of that methodology is centered around

delivering value-producing functionality quickly, and in a way

that anticipates significant change in direction. Much like the

software industry, BI has historically been plagued by constant

changes in requirements. Anecdotes abound of end-users

viewing a report for the first time and immediately responding

with new requirements. But despite this, legacy BI architecture

has failed to achieve any form of agility.

Executive Summary

This paper explains the fundamental assumption of

traditional BI platforms that was made when business

intelligence first emerged into the mainstream in the 1990s—

and why it is no longer valid. Given this false assumption, we

put forth the implications for how traditional platforms

operate, and how this compares to more agile, lightweight

platforms.

7900 Westpark Drive, Suite T-107 | McLean, VA 22102 | Toll Free: 1-888-LOGIXML | www.logixml.com | info@logixml.com

Traditional BI technologies, in assuming the goal of a single

version of the truth, have focused largely on overcoming

performance issues associated with meeting that goal. They

have done so through a heavy-weight process of transforming

data, dimensional modeling, summarizing or “cubing” data,

creating metadata layers, etc. This architecture builds in key

aspects of the decision process into every layer of the process.

The implication of this is that even relatively simple requirement

changes can trigger significant rework through the entire

architecture. For example, changes to source systems, ETL jobs,

the dimensional model, and the metadata often take months to

deliver.

When decisions are well known (“routine decisions”) such an

architecture can support them. The requirements for basic

financial reporting, for example, don’t change often. But many

times, we need to make decisions that are novel, such as what

markets to expand into, what products to introduce, or how to

respond to a new competitor. These types of decisions will often

require new information, and will often be very iterative in

nature. The way the decision is made changes as the decision

maker gains more information. And once made, such decisions

can change the landscape entirely. This is not a job for legacy

BI.

The monolithic approach of legacy BI has actually led to desktop

analysis tools (king of which is the spreadsheet) to become the

standard in such decisions. When Oracle decides to acquire

another BI vendor, that decision will be made in Excel, not

OBIEE. Why? Agility. Agility to change the decision model on the

fly. This monolithic architecture was required when 16-bit

computing and nascent relational database technology made

performance the primary barrier to decision support. With the

emergence of 64-bit computing, columnar databases, cloud

computing, and extreme data volume technologies like Hadoop,

legacy BI architecture needlessly sacrifices agility to solve last

century’s performance barriers. If BI is to be Agile, it must adopt

an architecture that assumes constant change in requirements at

all levels and is focused on the decision being made.

AGILITY: DRIVING CHANGE, NOT RESPONDING

David Weinberger, a senior researcher at Harvard’s Berkman

Center, talks about a phenomenon he calls the “changing shape

of knowledge.” The idea is very much at the root of why BI

requirements change so much, and in essence is a reflection that

as we learn more, we tend to change the way we view what we

knew in the first place. Most organizations struggle to keep up

with the changing shape of knowledge in the marketplace and

legacy BI likewise consistently fails to respond to these changes.

Weinberger studies the impact of the internet on society and the

impact of the internet in driving information globally has been

the key information technology success of the last century. The

internet was built on assumptions completely antithetical to

legacy BI, focusing on providing very focused information that

could be loosely joined to any other information stored anywhere

in the world. And the move into Web 2.0 has taken the web from

linking pages into a world in which we now are able to mash-up

rich applications.

Organizations have started to move away from wholesale

adoption of full ERP packages and back to a best-of-breed

approach supported by standards-based integration architectures

such as SOAP-based web services. Capabilities can be added,

changed, or removed without the need to completely re-architect

the entire system, thus providing agility to the business. This

architecture achieves the same type of flexibility that has made

spreadsheets proliferate, but provides a powerful framework to

avoid isolated, redundant and conflicting information silos.

Agile BI will follow this model, allowing domain and decision-

specific information applications to be joined together to form BI

platforms. These information applications will no longer be

stand-alone “enterprise BI systems,” but will often appear

embedded in the context of the transactional or other

applications already is use. And as requirements change,

applications will be added, removed, or updated quickly because

they don’t require an assessment of their impact on a universal

“single version of the truth” data model.

Agility is about driving these changes in the marketplace. Think

Wal-Mart or Amazon, but think paragons of Business Intelligence

who stretch their implementations beyond traditional views of BI

and use their insights to redefine and dominate their industry. In

order to obtain and maintain such competitive positions, such

organizations cannot wait for legacy architectures to catch up to

emerging requirements. When you are redefining the

competitive rules, you need decision support systems that can

keep up.

A NEW LANDSCAPE FOR BI

The environment in which any decision support system must

operate has completely transformed since the early attempts in

the 1970’s to create an EIS. While legacy BI architectures

continue to hold many of the same assumptions about

information and computing that were true in the early 1990s,

we’re seeing virtualization and cloud computing, Web 2.0

technologies, emerging standards and Services Oriented

7900 Westpark Drive, Suite T-107 | McLean, VA 22102 | Toll Free: 1-888-LOGIXML | www.logixml.com | info@logixml.com

Architectures, Advanced Analytics and Visual Analysis and a

variety of other innovations that have completely changed the

landscape.

The current imperative in BI is to abandon the assumptions that

have lead to such rigid solutions and leverage modern

approaches to decision support that provide greater agility for

the business. BI teams must move beyond legacy BI

architectures and include technologies that support a rapid,

iterative development style. The ability to rapidly source

information, connect it to other information in both a tightly and

loosely integrated fashion, and quickly connect BI applications

together will be critical in meeting rapidly changing

requirements.