Business intelligence Open source solutions - CELGE · source is also gaining market share in the...

White paper

___________________________

Business intelligence Open source solutions

2010 Edition

For further information: www.smile.fr

Tel.: +33 (0)1 41 40 11 00

Mailto: [email protected]

Business intelligence – Open Source solutions

© Copyright Smile - Open Source Solutions – All unauthorised reproduction is strictly forbidden

www.smile.fr

PREAMBLE

Smile

Smile is a company comprised of engineers specialising in the implementation of open source solutions and the integration of systems based on open source solutions. Smile is a member of APRIL, an association focused on the promotion and protection of free software.

With over 290 employees in France, and 320 throughout the world (September 2009), Smile is the leading French Open Source solution company.

Since around the year 2000, Smile has been actively monitoring the technological market, allowing us to identify, to test and assess the most promising open source solutions. We can then present our clients with the strongest, most sustainable, most efficient products available.

This approach has given way to a whole range of white papers covering various different application sectors. Content management (2004); portals (2005); business intelligence (2006); PHP frameworks (2007); virtualisation (2007); digital document management (2008); and ERPs (2008). Among the works published in 2009, the “Open Source VPNs”, and “Open Source flow controls and Firewalls” articles, within the “Systems and Infrastructures” collection are also of interest.

Each of these works offers a selection of the best open source solutions in the relevant domain, their respective qualities, and feedback on operational use.

As stable open source solutions slowly gain ground in new sectors, Smile will be present to offer customers the benefit of these solutions risk free. Smile appears in the French I.T. market as the integration service provider of choice, to assist major companies in adopting the best open source solutions.

Smile has also developed a range of service offers over the last few years. A consultancy department has assisted our clients since 2005, through preproject phases, solution research, and project support. In 2000, Smile created a graphics studio which in 2007 became known as The Interactive Media Agency. This agency offers not only graphic design services,



www.smile.fr

but also e-marketing, editorial, and rich interface expertise. Smile also has an agency specializing in Third-party Application Maintenance, application support and application processing. Smile offices can be found in Paris, Lyon, Nantes, Bordeaux and Montpellier, with presence in Spain, Switzerland, the Ukraine and Morocco.

Some Smile references

Web sites

Laboratoires Boiron, Foncia, Crédit Coopératif, EMI Music, Salon de l’Agriculture,

Mazars, Areva, Société Générale, Gîtes de France, Patrice Pichet, Groupama,

Eco-Emballage, CFnews, CEA, Prisma Pub, Véolia, NRJ, JCDecaux, Larousse,

01 Informatique, Spie, PSA, Boiron, Dassault-Systèmes, Action Contre la Faim,

BNP Paribas, Air Pays de Loire, Forum des Images, IFP, BHV, ZeMedical,

Gallimard, Cheval Mag, Afssaps, CNIL…

Portals and Intranets

Eurosport, HEC, Bouygues Telecom, Prisma, Veolia, Arjowiggins, INA, Primagaz,

Croix Rouge, Invivo, Faceo, Château de Versailles, Ipsos, VSC Technologies,

Sanef, Explorimmo, Bureau Veritas, Région Centre, Dassault Systèmes,

Fondation d’Auteuil, Korian, PagesJaunes Annonces, Primagaz…

Electronic Document Management and ECM

Agefiph, Primagaz, UCFF, Apave, Géoservices, Renault F1 Team, INRIA, CIDJ,

SNCD, Ecureuil Gestion, CS informatique, Serimax, Véolia Propreté, NetasQ,

Corep, Packetis, Alstom Power Services, Mazars…

E-business

Furet du Nord, Camif Collectivité, La Halle, De Dietrich, Adenclassifieds, Macif,

Gîtes de France, GPdis, Longchamp, Projectif, ETS, Bain & Spa, Yves Rocher,

Bouygues Immobilier, Nestlé, Stanhome, AVF Périmédical, CCI, Pompiers de

France, Commissariat à l’Energie Atomique…

Business Intelligence and ERP

Lafarge, Groupe Accueil, Anevia, Projectif, Xinek, Companeo, Advans, Point P,

Mindscape, Loyalty Experts, Cecim, Espace Loggia, Nouvelles Frontières,

France24, La Poste, HomeCineSolutions, Vocatis, Skyrock, France Domicile,

Polyexpert, Cadremploi, Cmonjob, Meilleurmobile.com…

Infrastructure and Hosting

Kantar, Pierre Audoin Consultants, Rexel, Motor Presse, OSEO, Sport24,

SETRAG, Canal-U, Institut Mutualiste Montsouris, ETS, Ionis, Osmoz, SIDEL,

Atel Hotels, Cadremploi, Institut Français du Pétrole, Mutualité Française…



www.smile.fr

This white paper

Having taken the lead in entire markets, such as content management, portals, or development frameworks, open source is also gaining market share in the business intelligence solutions sector, with solutions that are today highly-competitive.

As with the other white papers published by Smile, this work aims to bring together:

� A general Business Intelligence approach, concepts, fields of application, specific needs.

� A compilation of the best open source solutions in the intelligence domain.

� A relatively exhaustive presentation of these tools, their strengths, limits, maturity, ability to satisfy operational needs.

� A presentation on the existing professional aids to accompany these tools.

This study was based on several years of research and staff deployment, and constantly improved upon by way of operational implementation for our clients.



www.smile.fr

Version 2010

The first version of this white paper was edited in June 2006.

As a result of our project work and monitoring of tool progress, a new white paper version has been compiled every year since 2006.

This 2010 version includes information on what’s new in the open source solution market. As such we have a 2.4 version on SpagoBI, a 3.5.2 version on the Pentaho suite, and a 3.7 version on JasperSoft.



www.smile.fr

CONTENTS

PREAMBLE........................................................................................... 2

SMILE............................................................................................................................. 2 SOME SMILE REFERENCES ................................................................................................. 3 THIS WHITE PAPER ............................................................................................................ 4 2010 VERSION ................................................................................................................. 5

CONTENTS ........................................................................................... 6

PRINCIPLES ......................................................................................... 7

BUSINESS ANALYSIS TOOLS AND TECHNIQUES ........................................................................ 7 REPORTING ..................................................................................................................... 7 REQUEST OR AD HOC ANALYSIS ........................................................................................... 9 MULTIDIMENSIONAL ANALYSIS OR OLAP ............................................................................. 10 DATA CONSOLIDATION .................................................................................................... 14 ETL PRINCIPLES ............................................................................................................. 16 CONTROL PANELS ........................................................................................................... 17 DATA MINING ................................................................................................................. 18 TRENDS ........................................................................................................................ 18

BUSINESS INTELLIGENCE COMPONENTS ........................................... 20

PENTAHO DATA INTEGRATION : PDI (EX KETTLE).................................................................. 20 TALEND ........................................................................................................................ 25 BIRT ........................................................................................................................... 29 JASPER REPORTS ........................................................................................................... 32 PENTAHO REPORT DESIGNER............................................................................................ 35 PENTAHO ANALYSIS OR MONDRIAN .................................................................................... 38 JPIVOT ......................................................................................................................... 39 PALO ............................................................................................................................ 42 JPALO .......................................................................................................................... 44 WEKA........................................................................................................................... 46

BUSINESS INTELLIGENCE SUITES...................................................... 48

PENTAHO BI SUITE ......................................................................................................... 48 SPAGOBI ...................................................................................................................... 58 JASPERSOFT BI SUITE..................................................................................................... 64 PALO BI SUITE ............................................................................................................... 69

EDITOR SERVICE OFFERS ................................................................ 72

CONCLUSION...................................................................................... 73



www.smile.fr

PRINCIPLES

Business analysis tools and techniques

There was a drop in turnover in October. Decisions must be made to tackle the situation. However in order to make the right decision, research must be done to discover why

turnover had decreased and how it had decreased. What product range was affected? What countries, what regions? In the customer portfolios of which sales agent(s)? In which distribution segment? Do we have a similar drop every October? There are a number of questions to be asked and it is important to know how to respond to these questions in order to be in a position to make the right decision.

We can use a number of terms “business analysis tools and techniques”, or “decision support tools” or “business intelligence”, but basically we are referring to a number of I.T. solutions which allow us to analyze company data, in order to extract new reliable information on which decisions can be based, be these tactical or strategic.

A modern company stocks a huge volume of data on its information systems, however quite often, this very volume

makes it difficult to make sense of the data, and to understand

what the data represents: underlying trends, hidden strengths or weaknesses, everything we need to know to be able to make an informed decision.

As such, business analysis tools, before helping us to make a

decision, help us to aboard data analysis in order to identify

relevant macroscopic information hidden in the large volume of

data.

Report generator or Reporting

A report is the presentation of data in a coherent and legible manner, generally for printing.

The report displays number charts, managing headers, footers, subtotals etc.



www.smile.fr

A report generator is a program which allows one to define a report based on certain elements, with a minimum of programming, simply using an interactive interface. Once defined, the report can be generated at regular intervals.

A report can depend on settings: the year, month, region, sector of activity, etc. As such a single report which has been defined once can be generated using different variants, depending on the values of certain settings.

The report settings can be entered by the user based on their criteria, and the user can also vary these settings to narrow down their search.

In other cases, settings can be automatically defined by management rules, for example to give each sales agent a

report on their monthly sales. These are known as “burst

reports”.

There are two report generation phases:

� An initial conception phase, which usually requires specific expertise, and which is not in the hands of the end user;

� A settings and production phase, which does not require specialist expertise and can be carried out by the end user.

We would, of course, like to be able to limit the first phase, which is the most expensive, and be able to put the conception tools in the hands of the end user. But experience has shown that this is rarely possible: even though the user could learn how the tools work, they have other priorities.

In 2007 reporting tools for end users and “ad hoc” tools appeared in open source B.I. solutions. This allows end users with no specific technical capabilities, to generate simple reports easily and rapidly.



www.smile.fr

Request or Ad hoc analysis

There are two main business intelligence modes: the “strategic relations” mode and the “interactive mode”, with an analysis which interactively responds to user demands.

They each correspond to different needs and are sometimes complementary:

� Obtain recurrent information, which corresponds to predefined analysis, for example the breakdown of sales by region ;

� Searching for information by carrying out different successive analyses, the results of each lead to new questions, which in turn require the selection of different elements for analysis.

These tools are the same in some ways, but are quite different in others:

� The recurrent analyses are prepared upstream. Configuration can necessitate expert intervention. Once prepared, they will be produced regularly, almost automatically.

� On the other hand, interactive analyses must be under the end user’s control, who is no longer just an analysis consumer, but the key player, yet someone who is not a technical expert on business intelligence tools. This means that the tools used in these analyses must be very user friendly, both in regards to functions and to technical configuration. They must allow complete abstraction from technical know-how, and SQL database language in particular, as it must be assumed that users will not have prior technical knowledge.

� Finally, the interactive or ad hoc analysis also requires excellent response times, to allow the user to experiment directly and narrow down their analysis.



www.smile.fr

Multidimensional analysis or OLAP

Multidimensional analysis is one of the most commonly used methods of analysis in business intelligence.

Let’s try to outline the main principles.

We have basic information sets, generally in large volumes, each line of information represents an event characterised by:

� A unique identifier

� Elements that describe the information

� Values which carry information regarding quantities

Take the following standard example, which we have simplified for our purposes:

Take the basic entity the invoice line item. It is often a company’s key entity the most conclusive information it holds in regards to the sales process.

The invoice line item is part of an invoice and relates to the sale of a product, in a given quantity, at a given price, to a given customer.

It carries various different attributes: a product reference, a quantity, a unit price, a total price.

We can access other information via the invoice that the line item belongs to: the client reference, date (year, month, day) etc.

We also have access to other information on the customer themselves: country, region, type of customer, sector of activity, etc. The customer may even have an assigned sales agent.

We will leave it here for this example, but we could go a lot

further and collect a great deal of information all stemming from

this one little invoice line item.



www.smile.fr

invoice lineitem

customer sales agent

product category

country

invoice

The first step is to gather all of the information required for our analysis. Here, for example, the information required is the:

� Date (year, month, day)

� Product

� Product category

� Customer

� Customer country

� Sales agent

� Quantity

� Turnover

The first information constitutes the axes for potential

analyses, the last two constitute values.

In multidimensional analysis, the initial relational model is not the most pertinent or the most reliable. We generally prefer to denormalize the database, i.e. to work on a single table that all useful information has been gathered on.



www.smile.fr

In our example we would obtain the following table:

Date Product Segment Family Client Country Ccal Quantity T.O.

31/03/2007 12030991 Lawn

mower

Gardening Castorama France Lepaul 50 50 000

31/03/2007 14030952 Drill Tools Castorama France Lepaul 120 11 000

30/04/2007 12030993 Lawn

mower

Gardening LeroyMerlin Italy Legrand 250 230 000

Etc.…

There is clearly a great deal of redundancy in this table, but in the end it is easier to manage redundant but simple information, and to avoid problems regarding consistency and integrity which are an area for information systems.

This next step involves creating the first level of aggregation i.e. uniting certain line items.

In our case for example, we can assume that the data will not

be used at product reference level, but only by segment. In this case, we will regroup all of the identical lines for the key (date, segment, family, customer, country, sales agent), and add the quantity and turnover values.

The last step is that of “multidimensional analysis” in the true sense of the term, which consists of selecting the axes of analysis.

Among these axes, we can distinguish between:

� Discrete or discontinued value axes i.e. that contain a limited number of values, a postal code or CSP segment for example.

� Continuous value axes, typically a date or a price. We can lead them to a discrete number of values by defining ranges: a price range, an age range etc.

We can also distinguish between:

� Cumulable values, an amount or a number of items for example.

� Noncumulable values, age or date for example.

Cumulable values are those which it is wise to combine, i.e. those we can calculate the sum of, (or the average, or some other mathematical function of),

for a sub-total of lines, for each theme for example.



www.smile.fr

Sector

Month

01 02 03 04 05 06

NorthSouth

EastWest

Centre

Administrations

Health

Automotive

Tourism

Telecom

Bank

Section: Month 01Sample: month 06,

Eastern region

Turnover : €320KPeople : 12

Margin : €15K

CellMonth 04Tourism sectorEastern region

Multidimensional analysis consists in:

� Defining the axes of analysis that will be used, and the

order in which they will be used. E.g.: by region, then by

year, then by sales agent, then by product range. We do not always use all possible axes.

� Define the value(s) to be studied, and the function to be applied to these values.



www.smile.fr

Each hierarchy of axes of analysis correspond to a question that we ask it.

For example analysis by year, by sales agent aims at

representing the development of turnover, and comparing

sales agents for each preceding year. Analysis by sales agent,

by year aims to compare the sales agents first.

Data consolidation

Principles

The information or data which is the basis of the business intelligence system often comes from the company’s multiple different information systems. A business intelligence application often constitutes a datawarehouse, drawing on data from several sources within the company.

There are a number of reasons for this consolidation stage:

� Centralisation: it would be very difficult, sometimes even impossible, to access data on the various systems in real time: network interconnection problems, bandwidth problems, not to mention protocol and interface disparity issues.

� Unification: the datawarehouse gathers data in a single system, with a common repository, a unified model, and identical interface accesses. It allows to create links between data of mixed origins.

� Computer resources: business intelligence applications can take up a lot of CPU, disk space, memory, and the systems in place are not set up to be able to manage these new processes.

� Specialisation: business intelligence applications have specific needs which cannot be satisfied by the systems that companies have in place.

� The datawarehouse is a consolidation database, often a relatively standard relational database, but one which collects and manipulates large volumes of information. This could be an open source database (MySql, PostgreSql, etc.) or a commercial database (Oracle, SQL-Server).



www.smile.fr

ETL ETL ETLETL

HumanResourcesInformation

System

SalesManagementInformation

System

ProductionInformation

System

OthersInformationSystems

Datawarehouse

The datawarehouse centralises the business intelligence information derived from the companies various different information systems.

Organisation

As previously mentioned, this new database is not designed to enhance an operational application directly linked to the company’s activity (e.g. accounting system, H.R. database, sales agent database etc.). The datawarehouse is merely used to stock key company information and will be used purely for analysis and reporting query purposes.

This allows the company to analyse this data without diminishing the performance of common production tools.

The business intelligence database is modelled to facilitate “queries”. Two common forms of dimensional models are the snowflake and star schemas.



www.smile.fr

The tables are linked by only one key field in order to aid query performance, certain “value list” types of information will be repeated several times in dimensional tables.

As with the modelling of any database, this step is crucial to success of the project. The model must be flexible enough to allow future “data marts” to be received and to ensure the company real durability by archiving a large volume of information.

The datawarehouse must have a longer life expectancy than production tools. These tools can be enhanced or even replaced. In this case, the datawarehouse serves as the company’s memory, and only the load interfaces of this datawarehouse will be updated.

ETL Principles

We use a range of ETL “Extract, Transform, Load” tools to feed the datawarehouse. As the name indicates, these tools allow to extract data from different sources, to transform this data (format, name), and to load this data on the target database, here the datawarehouse.

The transformations entrusted to an ETL are often basic, but can in some cases include the procedural processes of specific programs.

With an ETL we avoid creating repetitive batch programs, often similar, the maintenance of which must also be ensured. The principle is that the integration of a new data flow does not require any development, and is operated by simple interactive configuration: we choose the data elements in the source repository, we specify the basic transformations to be carried out, and we indicate the data’s destination in the datawarehouse.

The ETL can manage different types of data sources, both at entry and exit point, the main ones being RDBMS and XML flows, but fixed format files or with comma-separated value (CSV) files are also managed.

Once an ETL flow has been defined, it is generally triggered on a regular basis; this is controlled by a task planning or scheduling tool.



www.smile.fr

An ETL generally treats point-to-point flows, i.e. from a single source to a single destination.

ETLs are designed to function in deferred-processing mode, most often at night. A datawarehouse saves a succession of “photos” of the activity of the company; the ETL allows to feed the datawarehouse with an additional “photo”. Once the feed is complete, the loaded data becomes statistics which are available to users.

Connectors

CSV files

DBMS (SQL)

XML

Connectors

CSV files

DBMS (SQL)

XML

Selection ofincoming fields

Formattransformationand other basic

procesings

Layout ofoutgoing fields

The dashboard

A dashboard is none other than a particular type of report.

A dashboard is a report that:

� Summarises information: everything must fit on one A4 page, or on one screenshot;

� Is most often a composite report, i.e. presents several indictors, which together offer an overall view of company activity;

� Is often also personalised, that is to say each user consults

a dashboard that relates to his/her activity.

Key Performance Indicators (KPI) are the indicators which are present from the home page of a business intelligence portal on.

Depending on the user needs, the dashboard may be sufficient on its own, or be the entry point to more precise analyses, of

secondary dashboards, or drill-down accesses.



www.smile.fr

The term Executive Information System or EIS is used in reference to a business intelligence system designed for use by the General Management; it contains highly-consolidated strategic information.

Data mining

Data mining refers to the search for useful statistical information in a large volume of data.

In multidimensional analysis reporting, the user knows what they are looking for: e.g. the breakdown of turnover by region. They don’t yet know what the breakdown is, but they do know what it consists of, and that this knowledge is relevant to their work.

In data mining, the user is looking for hidden statistical information which they haven’t yet been identified: trends, correlation, similarities, etc. Analysis of supermarket till receipts will typically show correlations between product purchases. While correlations some will have a simple explanations (razors & shaving gel), other will be more unexpected.

Such results need to be verified as the correlation observed may not be related to cause and effect but rather be just a coincidence, or the result of a hidden cause, both with phenomena’s such as conjoint consequences.

Data mining tools semi-automatically search through large volumes of data for these correlations & statistical rules, sometimes using a visual reproduction of these rules, which will reveal, for example, clusters of points on an axial representation of data.

Trends

Up to 2006 open source business intelligence was made up of Java components focused on specific functions: report edits, OLAP analysis, ETL, etc. These open source components were written and maintained by communities which have progressively evolved into the professional arena.



www.smile.fr

The current trend is clearly to professionalize open source business intelligence solutions.

The open source approach has become a “business model”, both already in place and also very promising, one in which investors are placing their money. Open source has made it possible to rapidly create key players with global ambition.

It is quite remarkable when we see that in 2008 and 2009, at a time when financers were rare and demanding, both Pentaho and JasperSoft were able to raise highly significant funds.

These editors offer complete business intelligence suites, the technology of which is under their control: the open source business intelligence environment is based on important suites which we analyse below.



www.smile.fr

BUSINESS INTELLIGENCE COMPONANTS

Before turning towards the creation of complete business intelligence solutions, open source projects each focus on a specific area of business intelligence.

As such, the BIRT and JasperReports projects allow to generate reports, while the Mondrian and Jpivot projects allow to present data in multidimensional format. These projects were and still are, designed to be integrated as “components” in specific developments.

Some open source business intelligence platforms are based on proven components and integrate these in order to constitute a uniform solution in which all functions are available in one place and made interoperable.

Here, we will present the main business intelligence components available in open source, which can be grouped into the following catagories:

ETL: Pentaho Data Integration (Kettle), Talend Open Studio

Report generators: BIRT, JasperReports & Pentaho Report Designer.

Analysis: Mondrian, JPivot, Palo & JPalo.

Data mining: Weka.

Pentaho Data Integration: PDI (formerly Kettle)

Introduction

PDI is an open source ETL which allows to design and execute data manipulation and transformation.

It’s visual step-based drag and drop environment makes it possible to create processes such as: data imports & exports, transformations such as: conversion, joining, application filtering, or even to run JavaScript functions, without having to generate and customise code.



www.smile.fr

PDI, under the name Kettle, became open source following version 2.2 and rapidly integrated the Pentaho business intelligence platform project (see below) which was then renamed Pentaho Data Integration.

Pentaho Data Integration version 3.1 is now available. The functional layout has been greatly improved and new steps have been added.

Features

PDI allows one to create two types of processes:

• Transformations: processes carried out at database level (one or more) including read, write, execute, operations.

• Tasks: processes of the highest level, combining actions such as the execution of a PDI transformation, mail dispatch, file downloads, or application launches. Different actions can be carried out based on the success of the previous step.

PDI can also be used with a shared repository. This allows several developers to use common objects. This repository is stored in a relational database and the developer can regularly connect to several repositories successively.

PDI can connect to a large number of databases, including Oracle, Sybase, MySQL, PostgreSQL, Informix, SQLServer and many more, and can also use data from text, XML and Excel files.

PDI has a:

� A file and read directory detector

� Read AND writing connector with Molap PALO cubes.

� LDAP connector and LDIF file reader

� SalesForce connector (read)

� List of available plug-ins at: http://wiki.pentaho.com/display/EAI/List+of+Available+Pentaho+Data+Integration+Plug-Ins

Transformations and tasks are created by drag and drop at different stages of the process. Assistants are available at each step and allow for example to automatically generate read and table creation queries.



www.smile.fr

An example of transformation

The applications

PDI is made up of 3 applications:

• Spoon: a visual environment where transformations and tasks are created and executed.

• Pan: online command application used to launch data transformation.

• Kitchen: online command application used to launch a data task.

Pan and Kitchen are used to plan the execution of transformations and tasks.

Installation of the extremely small Pan tool will allow the interfaces launched by PDI to be executed on several servers simultaneously. Here we are referring to execution in cluster mode. This allows you to control the operating times of your interfaces when your night “windows” are reduced.



www.smile.fr

Planning

It is generally preferable that the transformations or tasks created with PDI be executed periodically. This is the case for synchronisation or datawarehouse creation tasks in particular.

PDI does not directly integrate a task planner. Execution is based on an external system, such as a Windows or UNIX task planner.

Together with the Pentaho platform and via the Pentaho management console, the execution of PDI processes can be planned and monitored.

Use

Within the context of decision support, PDI serves to build a datawarehouse by importing data from different sources, consolidating this data and then formatting same.

Choosing PDI over a script-based solution offers several advantages:

• It is extremely easy to creation complex data flows

• Maintenance is simplified as all data flows are visible at the blink of eye, via a single centralised interface.

PDI can also be useful for projects that are not business intelligence related but that require the periodic execution of processes on different data or the synchronisation of different databases.

Example

Here we go through the various steps used to create and launch synchronisation between 2 tables stored in different databases:

1. Place an “Extract from table” object in the workspace.

2. Edit the “Extract from table” object, select the relevant data and table source. A SQL query is generated, which can be freely modified.

3. Place the “Insert in table” object in the workspace.



www.smile.fr

4. Link these 2 objects.

5. Edit the “Insert in table” object, select the data source and the name of the table. Kettle takes over creation of a new table or the modification of the existing table using the information provided in the previous steps.

6. Launch synchronisation.

Conclusion

PDI is an efficient tool to use to rapidly carry out all types of database processes. Necessary for most business intelligence projects, it can also be very useful for any project using databases and which requires synchronisation or export functions.

PDI has the advantage of being highly efficient, easy-to-use, entirely open source and interfaced with the Pentaho business intelligence suite presented below.

EXECUTION OF A TRANSFORMATION IN SPOON



www.smile.fr

Talend

Introduction

Talend1 is an open source ETL, developed by Talend based in France.

Talend is a “code generator” type ETL, i.e. it allows to graphically create manipulation and data transformation processes and to generate the corresponding executable in the form of a Java or Perl programme. This Java or Perl program must then be deployed on the execution server and run.

Talend is the ETL used by SpagoBI and JasperSoft (where it is known as JasperETL) solutions.

Defining processes

Data manipulation processes are created using a drag-drop graphical model. One can connect to the main databases and process different types of files (CSV, Excel, XML) using numerous steps. Talend has exclusive open source ETL features:

• The ELT steps (Extract Load Transform), which use the target database for different operations, thus hugely improving performance at no extra cost.

• Connectors for certain CRM applications (SugarCRM, SalesForces and CentricCRM), which avoids the manipulation of relational models of these tools.

It is worth noting that Talend simplifies the construction of database queries by detecting connections between tables thanks to foreign keys and an “Access” interface.

New features and components can be easily added to carry out more complex processes.

1 http://www.talend.com/



www.smile.fr

Business models

Talend also allows to create a business model in order to model interactions between the different systems and databases. Connection information, metadata, and documents

ALLOCATION AND MODIFICATION OF VALUES

PROCESSES



www.smile.fr

can be linked with each element. The model constitutes real documentation of the information system.

This mode is now complete with a technical documentation generator which allows developers to save time at the oft-forgotten technical specifications step.

Context management

With Talend one can manage contexts and apply them to transformations. The same process can, as such, be run in a development, test, or production environment, with the use of the right database connections and the right metadata.

Process deployment

Once the program has been generated, it is installed on the target machine by an administrator and its execution is

planned using the cron UNIX service or Windows task planner.

A “Distant Run” and “CPU Balancer” function is available with the TIS (Talend Integration Suite) version allowing Talend to be perfectly integrated into professional industrialised environments.

BUSINESS MODEL



www.smile.fr

Interface administrators have access to dashboards summarising the success or failure of processes, via the supervision console in Java client mode or in Web mode.

Conclusion

Talend is a comprehensive product. In developing partnerships with numerous business intelligence & database solution editors, Talend has strengthened its position in the market.

In 2008, Talend completed its range with a new tool “Talend Open Profiler” this tool further enhanced ETL activity by managing the quality of incoming data. This allows to control the unicity of data more easily, and to automatically recognise data from the same category.

It is worth noting that Talend also offers a suite “Talend Integration Suite (TIS)”, subject to annual subscription, which includes highly advanced features such as management of complex deployments, execution supervision, and the management of shared repositories.

As with PDI, Talend is of great use for business intelligence projects but is of even more use on projects relating to the urbanisation of information systems allowing to standardise exchange modes between the company’s different applications.



www.smile.fr

BIRT

Introduction

BIRT2 (The Business Intelligence and Reporting Tool) is an Eclipse community project, which includes a graphics generator, a report generator, and a design environment.

The project was initiated by Actuate3, a company that offers a commercial business intelligence range which integrates BIRT and which has adopted an open source license for BIRT.

Report generator

The BIRT engine is a library which generates HTML, PDF, XLS, DOC and PPT reports (configured or otherwise).

These reports can be complex containing several tables, charts, graphs and images. The data displayed emanates from different databases and queries.

The BIRT engine can be integrated into any Java developed application, be it a web or “heavy client” application.

2 http://www.eclipse.org/birt 3 http://www.actuate.com

EXAMPLE OF A BIRT REPORT



www.smile.fr

Report design

BIRT includes a report design tool which allows to create and modify reports graphically, by placing elements (tables, graphs) in a blank report. This tool can be integrated as a plug-in in the Eclipse4 development tool or can be used as a stand-alone application.

As, by definition, a report is simply an XML file, reports can be created without using this tool, but this takes much longer and is not recommended.

Reports are created in several steps:

• Sources of data are defined: one or more data sources (JDBC compatible databases or text files) can be added to a report.

• Data sets are defined: a data set is a selection of information which is formatted into a table or graph. It is the result of a simple or complex SQL query on a source of data. The user has an assistant to help with the creation of this query, and can combine diverse sources of assorted data by linking them on a common key.

• Element placement: the selected elements (tables, lists, graphs, images, etc.) are inserted into the report using “drag and drop”.

CRÉATION OF A BIRT REPORT



www.smile.fr

• Configuration: a dataset is associated with each element and the render settings are modified.

• Previewing: a preview option allows to launch report generation.

Target market

The BIRT report design tool is primarily for developers, as it requires the creation of data sets using SQL queries.

Having said that, BIRT can be made accessible to end users with a less “technical” profile, using certain features which circumvent the most complex steps:

Resource libraries containing basic elements can be created initially to create a new report: data sources, the company style sheet, and a selection of data sets. These elements can be incorporated in a new report using drag and drop.

The user can then create their own data sets by joining existing data sets. For example, if the library contains the data

set “clients” (a detailed list of clients ) and “sales” (a sales list including the client reference, product, etc), the user can

easily create a “sales per client” data set (a detailed sales list for each client). They can then add calculated columns and filter the results using the assistant.

Cross table element

Up to just a few months ago we could have accused BIRT of not being compatible with certain types of graphs or cross tables. This is no longer the case. Not only are GANTT, bubble, and other diagrams available, but BIRT now allows the user to easily create cubes and cross tables, from any data sets.

Conclusion

BIRT is an extremely comprehensive reporting tool, which allows to create complex reports quickly and to control their layout. It has the support of the dynamic Eclipse community.



www.smile.fr

Jasper Reports

Introduction

JasperReports4 is a reporting engine developed by JasperSoft5 and distributed on an open source license.

The generated reports are XML files and can also be created and modified manually.

These reports exist since 2001, and are already widely used in numerous business applications for their reporting attributes.

Report generator

The JasperReports engine generates reports in PDF, HTML, XML, CSV, RTF, XLS and TXT format. It uses JFreeChart to generate graphs and can be integrated into any Java developed application.

It is compatible with standard databases and multidimensional analysis servers allowing access to Mondrian server features directly in a JasperReport report.

Report created using JasperReport

4 http://jasperreports.sourceforge.net 5 http://www.jaspersoft.com



www.smile.fr

Report design

Reports are defined in XML file format or using a visual design tool. Naturally, we recommend use of the graphic interface.

Reports are divided into bands in which the graphic elements are placed. Each band has a specific behaviour and appears at least once.

Each report carries out iteration on a main data set. Certain bands are displayed before or after overall report data, others are displayed once for each dataset element.

Report Title

Page header

Column header

Detail

Column footer

Page footer

Last page

Summary

The different bands available are:

� Report title, displayed at the top of the first page.

� Page header, displayed at the top of each page.

� Column header, displayed before data.

� Detail, repeated for each data element.

� Column footer, displayed after each section of data.

� Page footer, displayed at the bottom of each page.

� Last page, displayed on the last page.

� Summary, report conclusion.



www.smile.fr

Secondary data sets can be used in certain elements to create richer reports (graphs, tables etc.).

iReport: Graphic interface

Several visual design tools can be used with JasperReports, but since the JasperSoft editor developed the business intelligence suite the iReport tool has supplanted all others including Jasper Assistant and JasperPal.

iReport6 is the official JasperReports design tool and is presented as a specific Java application.

It supports all JasperReports’ features and also has a specific administration extension for the JasperSoft business intelligence platform.

The JasperReport report interface and operating mode make iReport a tool suitable for specialists.

Conclusion

JasperReports is the report generator engine most widely used in the open source world.

All business intelligence platforms are compatible with JasperReports. Its main shortcoming is that the various visual

6 http://ireport.sourceforge.net

IREPORT



www.smile.fr

editors tools are not intuitive, which limits their use for developers.

Pentaho Report Designer

Introduction

JFreeReport joined the Pentaho project at the beginning of 2006. The name JFreeReport was slowly abandoned and replaced by Pentaho Report Designer (PRD).

PRD allows the user to develop complex reports and to publish them directly to the business intelligence server, using the Pentaho platform.

Creating reports

PRD supports PDF, HTML, CSV, Excel and Text formats.

Reports are only distributed and run in Web mode, with a mandatory display of the data of the day. Reports are never distributed and run by the end users using a heavy client; this avoids data refresh issues, as can be observed with other proprietary solutions.

The reports are defined in XML, basic programming allows them to be read and easily manipulated.



www.smile.fr

Report created using Pentaho Report Designer

Pentaho Report Designer is a visual design tool similar to iReport or Crystal Report, using the same notion of “bands”, for the repeated areas of the page (header, group header, column, footer, etc.).

Since version 1.7, PRD, now in version 3.5, has become considerably enriched and can be easily integrated into the suite with simplified web publication of reports.



www.smile.fr

From the home page of the user guide on, the user is walked through the report creation steps:

An assistant allows to establish a new connection but most importantly to also connect to a Pentaho metamodel (data dictionary).

This is an important breakthrough for open source reporting tools and in 2007 PRD was the first to allow the user with no SQL knowledge to create a report.

Several features are also available to carry out every-day operations such as counting occurrences in a group or format conversions. These features are displayed by theme in a library.



www.smile.fr

Conclusion

Pentaho Report Designer is an easy-to-use tool, which integrates well into the Pentaho business intelligence suite for management of settings or publication on a web platform.

It is unfortunate that cross tables cannot be created in PRD, unless pointing to an existing Mondrian schema. Nonetheless this feature has been announced in the editor’s 2010 roadmap.

Pentaho Analysis or Mondrian

Mondrian7 is an OLAP server (On Line Analytical Processing) or “OLAP engine” available under EPL open source licence (Eclipse Public Licence).

It is part of the “R-OLAP” server category, i.e. it can access data contained in a relational database.

Mondrian runs MDX queries, (a language also used by other OLAP engines such as that of Microsoft SQL Server). This language allows to create queries, the equivalent of which would necessitate a larger number of queries and take far more time to run in SQL language.

Mondrian is particularly powerful and optimises response times by using aggregate tables already created, but also

7 http://mondrian.sourceforge.net/



www.smile.fr

allows to make complex calculations, comparing elements on the time dimension, or by managing asymmetrical recursive hierarchies.

Mondrian is used with the clients JPivot, JPalo or Pentaho Analyzer (see below); tools which offer a graphic web interface to consult and manipulate data.

The Mondrian project joined the Pentacho project with

founder Julian Hyde, under the name of Pentaho Analysis. Mondrian is used by Pentaho and JasperSoft, and is maintained by Pentaho, with the participation of JasperSoft.

Pentaho recently acquired ‘Analyzer’ a new OLAP on Mondrian navigation interface, setting itself apart as regards ergonomics of the old Jpivot interface, still used by JasperSoft and SpagoBI.

JPivot

JPivot8 is an OLAP client with a Web interface. It presents OLAP cubes in multidimensional cross table format and allows users to perform standard analysis operations (drill down, drill up, rotations, filters, etc.) interactively.

JPivot also renders a graph corresponding to the data in the table, which is updated during exploration.

Tables and graphs can be exported in printable PDF file format or in Excel format so that the results obtained can be reused.

The user interface can be easily modified using style sheets (CSS and XSL).

8 http://jpivot.sourceforge.net/



www.smile.fr

JPivot table (web)

JPivot allows users to not only explore cubes, but also to carry out drill-throughs, i.e. allow users to view the raw data pertaining to a consolidated figure (e.g. The N lines in January invoices of product A in the R region).

Drill through in JPivot

JPivot is a reliable, easy-to-use interface which doesn’t require any specific configuration.



www.smile.fr

The Mondrian/JPivot union is available in the Pentaho Community Edition, JasperSoft, & SpagoBI suites.



www.smile.fr

Palo

Palo9 is a multidimensional in-memory database distributed under General Public licence (GPL).

Palo falls under the M-OLAP catagory: all data is loaded to the memory and calculated instantly, resulting in excellent response times.

The Palo database can be accessed, with a plugin, via a Microsoft Excel spreadsheet or OpenOffice Calc. This allows navigation of multidimensional data directly in the spreadsheets that users so like.

A PALO CUBE IN EXCEL

This integration allows to make use of native table and graph Excel features for the presentation layer, while the data displayed remains stored on the Palo database and not in Excel.

9 http://www.palo.net



www.smile.fr

MsExcel dashboard, created with Palo

Palo is not limited to Excel and has an API to Java and PHP languages. It can, as such, be queried using other applications such as Palo Eclipse Client (JPalo see below) or the other components of Palo, e.g. Palo Web, a spreadsheet in web mode.

Palo must use company data within the framework of a business intelligence project. The data must be imported into a Palo database, as opposed to Mondrian which works directly on a relational database. APIs are supplied and allow to write in the M-OLAP cube using C, Java, PHP and .NET languages. Furthermore, Jedox supply an ETL to feed the database, and the open source ETLs Pentaho Data Integration and Talend have connectors to PALO.

Data can also be imported from a text file via Excel or an ODBC connection, but this solution is obviously not adapted to the automated importing of a large volume of data.

Palo also offer a very interesting, unique feature: the capability of modifying values in a cube. Values can be generated in different statuses: budget value, real value, revised value, etc., this allows to build an application of decentralised entries and to consolidate the lot, or, alternatively, to enter global data and then break it down.

In its Enterprise edition, Palo also allows to transform an MsExcel entry or restore a template into web mode in just 3 clicks! This unique market feature, managed by the



www.smile.fr

Worksheet Server, is based on a Excel/.Net to PHP conversion server.

� It presents a template in lines/columns (as in Excel), (without rotation or drill capabilities).

� It allows, among others, to locally manage the Read-only mode on certain indicators and the Read-Write mode on others.

This feature allows administration of a controlled entry workflow in web mode: extremely useful not only for creating budgets, but also for centralising all types of business intelligence information in a controlled manner, via the web.

While Palo is not an “off the shelf” budgeting business tool, all relevant budgeting features are present, and we ourselves have successfully used it for this purpose.

JPalo

The JPalo10 project offers Palo server Java-developed tools, such as “Palo Eclipse Client” an OLAP client.

Palo Eclipse Client is a heavy client application which is integrated into Eclipse development environment. It allows to display Palo cube data in cross table format. The user can then carry out standard multidimensional analysis operations.

10 http://www.jpalo.net/



www.smile.fr

In the current version, Palo Eclipse Client does not offer presentation of data in graphic format.

JPalo also has a relatively easy-to-use web interface, which can be interfaced with different OLAP servers including Mondrian.

JPALO – WEB INTERFACE

Palo Eclipse Client



www.smile.fr

Weka

Introduction

Weka11 is a tool which allows to run data-mining algorithms on data. This makes it possible to isolate populations or extract rules from the data contained in the datawarehouse.

It is presented in the form of an independent application, with a graphic user interface or command line.

The user can apply different algorithms one by one, or create a workflow of processes, in a similar way to PDI, which could for example be used in a business intelligence platform to periodically analysis data.

Using Weka requires good knowledge of data-mining and the various statistics algorithms used.

WEKA - PRE-PROCESSING

11 http://www.cs.waikato.ac.nz/~ml/index.html



www.smile.fr

WEKA – DATA VIEW



www.smile.fr

BUSINESS INTELLIGENCE SUITES

For 3 years now, we have been studying and implementing the main open source business intelligence platforms.

Each solution is a completely separate software suite composed of the modules presented above, with a logic that is common to all.

While initially the suites shared certain open source components that we have looked at, and still share a certain number today, open source business intelligence solutions have maintained the tendency to differentiate themselves so that each can provide a solution which is best-adapted to each individual project.

Pentaho BI Suite

Introduction

Pentaho12 is a software suite which enables the distribution of business intelligence features and documents to a large number of users, via a Web interface.

Pentaho is available in Community or Enterprise edition. The Enterprise edition is subject to annual subscription and includes extra modules, product & project support and licence guarantees.

Pentaho is a clear forerunner in the open source solutions market. The editor has rallied the early days of open source products such as Kettle or Mondrian, and is well-capable of activating its community, directing it, and has the focus to base itself on community developments in order to complete them and integrate them into the enterprise edition of its solution.

12 http://www.pentaho.com



www.smile.fr

The dashboard designer is a good example of this anchorage in the open source community. This module is based on the “CFE” (Community Dashboard Framework), developed by the community for use by technicians. Using this framework as a base, the Pentaho team developed a WYSIWYG web mode configuration interface, this time directed at business users. The end-user module is now integrated into the enterprise edition.

Features

Pentaho allows access to the various business intelligence components of the suite via a single interface.

This makes it possible to consult reports (created with BIRT or Pentaho Report Designer), to use the multidimensional exploration features of Mondrian, and to create dashboards.

When a document is opened the user can modify the execution settings, by choosing the relevant period or the products presented, for example.

The platform also allows to automatically send a group of reports to their respective recipients, by email. Dispatch can occur either periodically or on occasion.

User interface

The features of the Pentaho platform can be used in a web application, a portal or by calling on web-services.

Pentaho includes two interfaces for the presentation and execution of the different actions available.

♦ Web application

The Pentaho default interface gives easy access to all available actions and administration features.

Security is highly regulated, at every level: document access, permissions to an action or document and data security depending on the person connected, wherein the manager of country R, for example, only has access to data on country R and not other countries.



www.smile.fr

PENTAHO USER CONSOLE

This interface constitutes the easiest way of implementing the Pentaho platform and as such it is often used first.

♦ Portal

Since the 2.0 version and the availability of a user-friendly, multi-tab, user console, the integration of Pentaho into a portal has become less important, pointless even. It can still be done however, either using iframes in Pentaho, an operating mode which can be duplicated for integration into a portal; or by the availability of standard portlets which allow the display of report lists and then to run them.

The integration of Pentaho into a portal also requires the implementation of a Single Sign On system, with user management in an external repository, usually a LDAP directory and the sharing of tokens, via the open source solution CAS for example.



www.smile.fr

Figure 1 : Pentaho report in a Liferay portal

In this mode, the security and customisation of the interface are managed by the portal. This makes it possible, for example, to choose which portlets are displayed for each user. As with the “standard” interface user information is propagated up to Pentaho actions, which allows, to return adapted, profiled documents.

The portlets supplied by Pentaho are used to integrate the platform in existing company portals or intranets.

Workflow engine

The workflow engine used is “Enhydra Shark”13. Its role is to structure the execution of business intelligence processes:

Each function (report generation, analysis) is referred to as the execution of an action or “Xaction” in Pentaho-speak.

An xaction is a succession of simple steps, such as the execution of a query on a database, the generation of a status report, sending a mail or the execution of a PDI job. The processes carried out by these actions can be quite elaborate, as it is possible to process each element of an overall result separately, or to carry out specific calculations by running a JavaScript program.

13 http://www.enhydra.org/workflow/shark/index.htm



www.smile.fr

The workflow engine manages the continuation of one step on to another.

Each xaction is described by a simple XML file. The Pentaho Pentaho Design Studio tool can be integrated into the Eclipse development environment and allows files to be created graphically.

Pentaho Design Studio

All of the business intelligence components supported by Pentaho constitute a workflow step, which allows their

SENDING A REPORT BY MAIL



www.smile.fr

features to be combined, and to easily extend the platform possibilities by developing new steps, to make it, if you so wish, a real business application.

Planner

“Quartz”14 is the planner used. It allows to plan any Pentaho action.

With Quartz the user can create an action which generates and sends a status email, and plan routine execution.

Planner actions (add, suspend, delete), can be controlled either manually or by following the steps which correspond to an action.

Pentaho allows not only to use business intelligence tools with a single interface, but also to combine their functions and to enrich them via processes carried out before and after their execution.

Metadata

One of the important features of Pentaho suite is the Pentaho Metadata Editor, which allows to create database “business views” that can be read by end users.

The user no longer sees the database tables directly but grouped information, translated into French (or English, or several languages depending on the user), potentially resulting from several tables and displayed together.

Once these business views have been defined, “Web Ad hoc Query” a web interface allows query creation, without use of SQL language, by selecting on the screen, the fields which are to be displayed and the groupings to be carried out.

The Metadata Editor saves the model in “Common Warehouse Metamodel” format, an open standard (unfortunately seldom shared) of the warehouse description.

14 http://www.opensymphony.com/quartz/



www.smile.fr

Pentaho Metadata Editor

Analyzer

Analyzer is a new multidimensional navigation web interface for the Mondrian server.

This wasn’t developed by the Pentaho teams, but Pentaho acquired it from the Editor LucidEra. LucidEra did offer its software in SaaS, but unfortunately the company went bankrupt. This acquisition proved to be truly exceptional, the LucidEra tool was already compatible with the Pentaho Mondrian server allowing integration of the GA version in less than a month, and on top of that the product developers also decided to join Pentaho.

Analyzer allows the user to build a multidimensional view by dragging and dropping dimensions & indicators, and to then filter and exclude values. This allows business users to create calculated fields by entering formulas, making the possibilities endless.



www.smile.fr

Pentaho Analyzer

With this Analyzer, Pentaho is decidedly oriented towards the end user, offering an analysis tool which is simple to access and very powerful.

Dashboard designer

Since version 3 Pentaho now offer a “dashboard designer”, this allows the creation of composite pages in web mode. Pentaho choose to use a layout with predefined models in which zones can contain existing reports or new ad hoc queries.

In each zone, the user can choose to display a table, graph, cube, report, etc. The tables and graphs can be created from an ad hoc requester, and we particularly like the quality of the graphics, created using flash.



www.smile.fr

The user can also define filters which can be applied to all zones and obtain a dashboard like the one below:

Community

The Pentaho project is currently highly active and displaying the results of a leading software editor.

Several open source business intelligence projects such as Kettle, Mondrian, Weka and JFreeReport have joined the Pentaho project; this guarantees perfect integration and a coherent evolution of the whole.

The community around the project is also very dynamic as reflected by the activity on the Pentaho forum and the various contributions made to the project by users.



www.smile.fr

Conclusion

Pentaho version 3.5 is an extremely comprehensive and self-sufficient business intelligence platform.

It not only allows use of the different open source business intelligence tools via a single user-friendly interface, but also allows to extend and combine their features with the use of a workflow engine.



www.smile.fr

SpagoBI

Introduction

SpagoBI15 is a business intelligence platform developed by the Italian engineering company Ingegneria Informatica16.

This platform is distributed exclusively under open source GPL.

Features

SpagoBI allows the user to run JasperReport and BIRT reports. SpagoBI also permits data analysis with Mondrian/JPivot.

SpagoBI includes a query creation help component (Query By Example) which allows the user to graphically select the production database data (not necessarily from a datawarehouse) to be displayed and to then sort, group, select and export the result.

The SpagoBI graphic interface used the portal eXo portal17 in version 1.x, which added an extra load to platform administration, but version 2.x allows to do away with this, while its integration with a portal remains possible.

15 http://spagobi.eng.it 16 http://www.eng.it/ 17 http://www.exoplatform.com/



www.smile.fr

User interface

JPIVOT IN SPAGOBI

The solution takes charge of security, with management of user permissions regarding business intelligence pages and groups. Different profiles can be set up for users, developers and administrators.

BIRT report in SpagoBI



www.smile.fr

NAVIGATION IN THE DOCUMENT TREE STRUCTURE

Publication workflow

SpagoBI integrates management of the creation, test, and validation cycle for business intelligence documents (reports, multidimensional cubes, etc.). All of these operations are carried out directly from the administration interface.

Report “drafts” which only the developers can see and test, can also be created, without these appearing in the user environment.

The different versions of each report and each attachment are saved allowing the developer can go back to older versions at any time.

This feature allows business intelligence document updates and testing in real conditions prior to production, without having to access the server itself.

Useful components

SpagoBI uses several open source business intelligence components:

• Reports: JasperReports, BIRT



www.smile.fr

• Analysis: Mondrian, JPivot, JPalo

• Data-mining: Weka

• ETL: Talend Open Studio

The SQL query creation feature (Query By Example) was specially developed by SpagoBI, as was the KPI display engine.

KPI ENGINE

Data integration

SpagoBI choose Talend as the ETL of reference for this platform. Data processes can be planned and triggered directly from the portal.

SpagoBI Studio and Meta

In late 2009, the SpagoBI teams delivered a SpagoBI Studio version, a development environment for reports and configuration of the various engines available in SpagoBI.

Integrated into Eclipse, the Studio allows report creation with BIRT or JasperReport, and then to deploy it on the SpagoBI web platform with a simple click, where before one had to upload the generated file to the web platform and then create value lists.



www.smile.fr

The Studio also makes it easier to create composite dashboards (geographic maps, cubes, etc.).

This is most definitely an important breakthrough, which considerably lightens the workload involved in creating/configuring business intelligence documents in SpagoBI.

SpagoBI Meta

Meta is a semantic layer on which SpagoBI is based for its reporting engines. Though it is still in Alpha version Meta already allows the creation of “business models” via the Eclipse environment, on which the QbE interface then allows to user to make queries and reports.

It is still a new component, but one which will develop very quickly in the coming months, and which will in the end fill a SpagoBI functional need and allow users to do their own reporting, autonomously.

Community

The SpagoBI project community remains smaller than that of Pentaho, but its OW218 membership has created a dynamic and allows the community to share with other open source modules and editors.

18 http://www.ow2.org



www.smile.fr

Conclusion

SpagoBI is a complete business intelligence platform with a selection of business intelligence components either developed in-house: KPI, Maps, QbE, or by a third party: Mondrian/JPivot, BIRT, JasperReport, Weka, Microsoft SSRS etc. There is even a Business Objects reports connector available.

SpagoBI were wise to offer business intelligence system management features right from the beginning, with versioning, workflows, comments, contextualisation, management of user permissions, etc. which make it a very interesting tool and one which is very practical to use.



www.smile.fr

JasperSoft BI Suite

Introduction

JasperServer19 is the JasperSoft business intelligence platform, (JasperSoft also developed the report generator JasperReport, available since 2001).

This platform offers reporting and analysis features and is available under two licences: GPL and commercial.

Features

In JasperServer we find all of the JasperReport and JPivot/Mondrian features already outlined in this document. Users can plan report generation themselves and either receive the document by mail, or store it in the repository directory.

The interface does not use third party portals, as opposed to other platforms. Here we have a specially developed web application which allows to create new analyses, to publish new documents, manage user accounts, roles, and access permissions to access different directories and repository files.

19 http://www.jaspersoft.com/

VIEWING A REPORT IN JASPERSERVER



www.smile.fr

NOTE: for administrators; reports can be published and modified directly from iReport, the JasperReport report design application.

VIEWING A BUSINESS INTELLIGENCE REPOSITORY IN JASPERSOFT

Components used

• Reports: JasperReports

• Analyse: Mondrian and JPivot, under the JasperAnalysis name

• Planner: Quartz

• ETL: Talend Open Studio, under the Jasper ETL name.

It is unlikely that this platform will support other report generators in the future as it is in JasperSoft’s best interest to promote its own reporting tool.

Domains

JasperSoft allows the creation of domains (semantic layers above relational SQL databases which allow to define business glossaries, and allow users to create reports without having to understand or write SQL).

One of the special features of JasperSoft is the domain creation feature in web mode, which makes them very accessible. While the tool may not be visual, its remains very



www.smile.fr

practical and offers several highly advanced features, such as the creation of derived tables or complex calculated fields.

Creation of a domain

Ad hoc analysis for business users

JasperServer professional and enterprise editions include additional features not included in the community project edition.

One of the main differences is the online ad hoc report design tool (lists, graphs or cross tables) which is accessible to all professional & enterprise edition users.

This allows end users to create new web mode documents in just a few minutes, using a visual interface based on assistants and drag/drop.



www.smile.fr

The cross table editor is of particular interest. First of all, it allows the creation of cubes in memory on any relational database, without having to go through an OLAP server. It also allows user to create their own filters, calculated values and other standard actions. Finally, the periodic execution of this cube can be programmed with, for example, a mail dispatch. Not forgetting that to simplify the manipulation, Jasper allows the user to work solely on an extract of values, i.e. 100 lines instead of the entire table.

Cross table with JasperServer ad hoc

There are greater limitations than with a “real” cube, as can be seen at aggregate management level, not managed by Jasper



www.smile.fr

for the moment, or at indicator calculation level - where temporal calculations are difficult to implement.

The report creation web interface is a real bonus, allowing end users with no technical knowledge, to independently build their analysis from data made available.

Dashboard design

JasperSoft also include a dashboard designer tool, accessible to all users.

The layout is free and created by dragging and dropping existing reports or analyses. Filters can be added easily and the tool suggests different filters for selected elements.

Jasper WorkBench

JasperSoft supplies a Mondrian cube design application. The application uses a simple extra layer which relates to the manual edition of XML files: each tag can be modified with a basic form.

This application is not invaluable, but does help developers to save time by avoiding XML tag errors and by suggesting database table names and associated columns at the right time.



www.smile.fr

Conclusion

JasperSoft has developed neither its own data mining tool nor ETL, distributing that of Talend. It has, however, concentrated on the usability of its solution for the end user. The results are excellent, with 2.0 ergonomics and great autonomy in the creation of reports and new dashboards for non-tech users.

Palo BI Suite

The German company Jedox edits the PALO multidimensional database together with different related components making this a comprehensive business intelligence suite.

Schema Workbench



www.smile.fr

Web navigator

Palo for Excel

Jedox offers a plugin for Microsoft Excel which functions in read and write mode.

There is also the “supervision server” which allows to follow the modifications made by users in Excel cubes and to generate an action. The “worksheet server” allows to distribute the views carried out with Excel directly in web mode.

Palo Web

In version 3.0 Jedox developed a spreadsheet from scratch in web mode, the impressive result is a spreadsheet that is very similar to Excel.



www.smile.fr

Palo Web

There are multiple advantages to a web spreadsheet. For one the distribution of a dashboard to its recipients is simplified and immediate, furthermore Excel limitations such as the locking of cell formulas for entry zones or the formatting of repeated cross tables are overcome.

Other important features are also available around the web spreadsheet which surpass the simple Excel client, the report and template repository for example.

One of the strengths of Palo remains the Excel interface; the web platform remains compatible with MsExcel and allows the importation of files worked on MsExcel to the web.

Conclusion

The range of features included with Palo, and the technological choices associated with it, are unique and meet different project needs to the other open source suites. Palo is often the only open source tool companies use for their business intelligence projects, though it can also be used to complement solutions such as Pentaho or Jasper with their ad hoc reporting capabilities.

Since 2008 Jedox has an office situated in France, this has speeded up its deployment in France.



www.smile.fr

SERVICES OFFERED BY THE EDITORS

If the development of a specific technical tool in the heart of a community has a sense, it is that each person can use the tool in question and add their contributions, the development by a community of a business intelligence suite does not exist or does not exist YET, much as there is a huge effort made.

It is real commercial companies that are behind the various different business intelligence open source suites presented in this document.

These companies, apart from sharing – or fighting over –components developed by certain communities, try to find the best means of securing a return of their investment in this new business of open source editor solutions.

Apart from SpagoBI which distributes the entire software solution under GPL, the other editors (Pentaho, Jasper, Palo, Talend) have all chosen to make a free limited version available, and to reserve additional modules for the enterprise edition.

On top of these extra modules, the enterprise editions include editor support, with not only standard software anomaly support but solution implementation assistance. In any case, the editor support is guaranteed in the case of a problem, giving you access to editor expertise so you are not at a standstill regarding the project or software exploitation.

The support contracts also generally include an “intellectual property protection” clause, offering legal assurance which protects the client from any patent legal action that may be taken. A clause which is greatly appreciated in the U.S. and U.K.



www.smile.fr

CONCLUSION

Open source business intelligence platforms are now considered comprehensive and credible alternatives to commercial business intelligence solutions. The various different components used exist for a number of years, are tried & tested, and highly reliable.

The web is the preferred support for open source platforms, offering visualisation, report design and data analysis.

While solutions such as Pentaho, Jasper and SpagoBI allow implementation of a complete business intelligence portal, business intelligence features can easily be added to an existing application, and the components on which these platforms are based be used separately.

Since the first version of this white paper, tools have been constantly developing and now integrate powerful interactive features, much-appreciated by users and administrators alike.

Numerous companies now choose to use open source business intelligence tools, benefiting from their performance, scalability, capacity to adapt to existing solutions, and their reduced cost.

Business intelligence Open source solutions - CELGE · source is also gaining market share in the...

Documents

Transcript of Business intelligence Open source solutions - CELGE · source is also gaining market share in the...