Horizontal aggregation for SQL

Horizontal Aggregations in SQL to PrepareData Sets for Data Mining Analysis

Carlos Ordonez and Zhibo Chen

Abstract—Preparing a data set for analysis is generally the most time consuming task in a data mining project, requiring many

complex SQL queries, joining tables, and aggregating columns. Existing SQL aggregations have limitations to prepare data sets

because they return one column per aggregated group. In general, a significant manual effort is required to build data sets, where a

horizontal layout is required. We propose simple, yet powerful, methods to generate SQL code to return aggregated columns in a

horizontal tabular layout, returning a set of numbers instead of one number per row. This new class of functions is called horizontal

aggregations. Horizontal aggregations build data sets with a horizontal denormalized layout (e.g., point-dimension, observation-

variable, instance-feature), which is the standard layout required by most data mining algorithms. We propose three fundamental

methods to evaluate horizontal aggregations: CASE: Exploiting the programming CASE construct; SPJ: Based on standard relational

algebra operators (SPJ queries); PIVOT: Using the PIVOT operator, which is offered by some DBMSs. Experiments with large tables

compare the proposed query evaluation methods. Our CASE method has similar speed to the PIVOT operator and it is much faster

than the SPJ method. In general, the CASE and PIVOT methods exhibit linear scalability, whereas the SPJ method does not.

Index Terms—Aggregation, data preparation, pivoting, SQL.

Ç

1 INTRODUCTION

IN a relational database, especially with normalized tables,a significant effort is required to prepare a summary data

set [16] that can be used as input for a data mining orstatistical algorithm [17], [15]. Most algorithms require asinput a data set with a horizontal layout, with severalrecords and one variable or dimension per column. That isthe case with models like clustering, classification, regres-sion, and PCA; consult [10], [15]. Each research disciplineuses different terminology to describe the data set. In datamining the common terms are point-dimension. Statisticsliterature generally uses observation-variable. Machinelearning research uses instance-feature. This paper intro-duces a new class of aggregate functions that can be used tobuild data sets in a horizontal layout (denormalized withaggregations), automating SQL query writing and extend-ing SQL capabilities. We show evaluating horizontalaggregations is a challenging and interesting problem andwe introduce alternative methods and optimizations fortheir efficient evaluation.

1.1 Motivation

As mentioned above, building a suitable data set for datamining purposes is a time-consuming task. This taskgenerally requires writing long SQL statements or custo-mizing SQL code if it is automatically generated by sometool. There are two main ingredients in such SQL code: joinsand aggregations [16]; we focus on the second one. Themost widely known aggregation is the sum of a columnover groups of rows. Some other aggregations return the

average, maximum, minimum, or row count over groups ofrows. There exist many aggregation functions and operatorsin SQL. Unfortunately, all these aggregations have limita-tions to build data sets for data mining purposes. The mainreason is that, in general, data sets that are stored in arelational database (or a data warehouse) come from OnlineTransaction Processing (OLTP) systems where databaseschemas are highly normalized. But data mining, statistical,or machine learning algorithms generally require aggre-gated data in summarized form. Based on current availablefunctions and clauses in SQL, a significant effort is requiredto compute aggregations when they are desired in a cross-tabular (horizontal) form, suitable to be used by a datamining algorithm. Such effort is due to the amount andcomplexity of SQL code that needs to be written, optimized,and tested. There are further practical reasons to returnaggregation results in a horizontal (cross-tabular) layout.Standard aggregations are hard to interpret when there aremany result rows, especially when grouping attributes havehigh cardinalities. To perform analysis of exported tablesinto spreadsheets it may be more convenient to haveaggregations on the same group in one row (e.g., to producegraphs or to compare data sets with repetitive information).OLAP tools generate SQL code to transpose results (some-times called PIVOT [5]). Transposition can be more efficientif there are mechanisms combining aggregation andtransposition together.

With such limitations in mind, we propose a new class ofaggregate functions that aggregate numeric expressions andtranspose results to produce a data set with a horizontallayout. Functions belonging to this class are called horizontalaggregations. Horizontal aggregations represent an ex-tended form of traditional SQL aggregations, which returna set of values in a horizontal layout (somewhat similar to amultidimensional vector), instead of a single value per row.This paper explains how to evaluate and optimize horizontalaggregations generating standard SQL code.

678 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 24, NO. 4, APRIL 2012

. The authors are with the Department of Computer Science, University ofHouston, Houston, TX 77204. E-mail: {ordonez, zchen6}@cs.uh.edu.

Manuscript received 17 Nov. 2009; revised 2 June 2010; accepted 16 Aug.2010; published online 23 Dec. 2010.Recommended for acceptance by T. Grust.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log Number TKDE-2009-11-0789.Digital Object Identifier no. 10.1109/TKDE.2011.16.

1041-4347/12/$31.00 � 2012 IEEE Published by the IEEE Computer Society

http://www.ieeexploreprojects.blogspot.com

1.2 Advantages

Our proposed horizontal aggregations provide severalunique features and advantages. First, they represent atemplate to generate SQL code from a data mining tool. SuchSQL code automates writing SQL queries, optimizing them,and testing them for correctness. This SQL code reducesmanual work in the data preparation phase in a data miningproject. Second, since SQL code is automatically generated itis likely to be more efficient than SQL code written by an enduser. For instance, a person who does not know SQL well orsomeone who is not familiar with the database schema (e.g.,a data mining practitioner). Therefore, data sets can becreated in less time. Third, the data set can be created entirelyinside the DBMS. In modern database environments, it iscommon to export denormalized data sets to be furthercleaned and transformed outside a DBMS in external tools(e.g., statistical packages). Unfortunately, exporting largetables outside a DBMS is slow, creates inconsistent copies ofthe same data and compromises database security. There-fore, we provide a more efficient, better integrated and moresecure solution compared to external data mining tools.Horizontal aggregations just require a small syntax exten-sion to aggregate functions called in a SELECT statement.Alternatively, horizontal aggregations can be used togenerate SQL code from a data mining tool to build datasets for data mining analysis.

1.3 Paper Organization

This paper is organized as follows: Section 2 introducesdefinitions and examples. Section 3 introduces horizontalaggregations. We propose three methods to evaluatehorizontal aggregations using existing SQL constructs, weprove the three methods produce the same result andwe analyze time complexity and I/O cost. Section 4presents experiments comparing evaluation methods, eval-uating the impact of optimizations, assessing scalability,and understanding I/O cost with large tables. Related workis discussed in Section 5, comparing our proposal withexisting approaches and positioning our work within datapreparation and OLAP query evaluation. Section 6 givesconclusions and directions for future work.

2 DEFINITIONS

This section defines the table that will be used to explainSQL queries throughout this work. In order to presentdefinitions and concepts in an intuitive manner, we presentour definitions in OLAP terms. Let F be a table having asimple primary key K represented by an integer, p discrete

attributes, and one numeric attribute: F ðK;D1; . . . ; Dp; AÞ.Our definitions can be easily generalized to multiplenumeric attributes. In OLAP terms, F is a fact table withone column used as primary key, p dimensions and onemeasure column passed to standard SQL aggregations. Thatis, table F will be manipulated as a cube with p dimensions[9]. Subsets of dimension columns are used to group rowsto aggregate the measure column. F is assumed to have astar schema to simplify exposition. Column K will not beused to compute aggregations. Dimension lookup tableswill be based on simple foreign keys. That is, one dimensioncolumn Dj will be a foreign key linked to a lookup table thathas Dj as primary key. Input table F size is called N (not tobe confused with n, the size of the answer set). That is,jF j ¼ N . Table F represents a temporary table or a viewbased on a “star join” query on several tables.

We now explain tables FV (vertical) and FH (horizontal)that are used throughout the paper. Consider a standardSQL aggregation (e.g., sum()) with the GROUP BY clause,which returns results in a vertical layout. Assume there arejþ k GROUP BY columns and the aggregated attribute is A.The results are stored on table FV having jþ k columnsmaking up the primary key and A as a nonkey attribute.Table FV has a vertical layout. The goal of a horizontalaggregation is to transform FV into a table FH with ahorizontal layout having n rows and jþ d columns, whereeach of the d columns represents a unique combination ofthe k grouping columns. Table FV may be more efficientthan FH to handle sparse matrices (having many zeroes),but some DBMSs like SQL Server [2] can handle sparsecolumns in a horizontal layout. The n rows representrecords for analysis and the d columns represent dimen-sions or features for analysis. Therefore, n is data set sizeand d is dimensionality. In other words, each aggregatedcolumn represents a numeric variable as defined instatistics research or a numeric feature as typically definedin machine learning research.

2.1 Examples

Fig. 1 gives an example showing the input table F , atraditional vertical sum() aggregation stored in FV , and a

horizontal aggregation stored in FH . The basic SQLaggregation query is:

SELECT D1; D2,sum(A)

FROM F

GROUP BY D1; D2

ORDER BY D1; D2;

ORDONEZ AND CHEN: HORIZONTAL AGGREGATIONS IN SQL TO PREPARE DATA SETS FOR DATA MINING ANALYSIS 679

Fig. 1. Example of F , FV , and FH .


Notice table FV has only five rows because D1 ¼ 3 andD2 ¼ Y do not appear together. Also, the first row in FV hasnull in A following SQL evaluation semantics. On the otherhand, table FH has three rows and two (d ¼ 2) nonkeycolumns, effectively storing six aggregated values. In FH itis necessary to populate the last row with null. Therefore,nulls may come from F or may be introduced by thehorizontal layout.

We now give other examples with a store (retail) databasethat requires data mining analysis. To give examples ofF , wewill use a table transactionLine that represents the transactiontable from a store. Table transactionLine has dimensionsgrouped in three taxonomies (product hierarchy, location,time), used to group rows, and three measures representedby itemQty, costAmt, and salesAmt, to pass as arguments toaggregate functions.

We want to compute queries like “summarize sales foreach store by each day of the week”; “compute the totalnumber of items sold by department for each store.” Thesequeries can be answered with standard SQL, but addi-tional code needs to be written or generated to returnresults in tabular (horizontal) form. Consider the followingtwo queries:

SELECT storeId,dayofweekNo,sum(salesAmt)

FROM transactionLine

GROUP BY storeId,dayweekNo

ORDER BY storeId,dayweekNo;

SELECT storeId,deptId,sum(itemqty)FROM transactionLine

GROUP BY storeId,deptId

ORDER BY storeId,deptId;

Assume there are 200 stores, 30 store departments, andstores are open 7 days a week. The first query returns1,400 rows which may be time consuming to compare witheach other each day of the week to get trends. The secondquery returns 6,000 rows, which in a similar manner, makesdifficult to compare store performance across departments.Even further, if we want to build a data mining model bystore (e.g., clustering, regression), most algorithms requirestore id as primary key and the remaining aggregatedcolumns as nonkey columns. That is, data mining algo-rithms expect a horizontal layout. In addition, a horizontallayout is generally more I/O efficient than a vertical layoutfor analysis. Notice these queries have ORDER BY clausesto make output easier to understand, but such order isirrelevant for data mining algorithms. In general, we omitORDER BY clauses.

2.2 Typical Data Mining Problems

Let us consider data mining problems that may be solvedby typical data mining or statistical algorithms, whichassume each nonkey column represents a dimension,variable (statistics), or feature (machine learning). Storescan be clustered based on sales for each day of the week. Onthe other hand, we can predict sales per store departmentbased on the sales in other departments using decision treesor regression. PCA analysis on department sales can revealwhich departments tend to sell together. We can find outpotential correlation of number of employees by gender

within each department. Most data mining algorithms (e.g.,clustering, decision trees, regression, correlation analysis)require result tables from these queries to be transformedinto a horizontal layout. We must mention there exist datamining algorithms that can directly analyze data setshaving a vertical layout (e.g., in transaction format) [14],but they require reprogramming the algorithm to have abetter I/O pattern and they are efficient only when theremany zero values (i.e., sparse matrices).

3 HORIZONTAL AGGREGATIONS

We introduce a new class of aggregations that have similarbehavior to SQL standard aggregations, but which producetables with a horizontal layout. In contrast, we call standardSQL aggregations vertical aggregations since they producetables with a vertical layout. Horizontal aggregations justrequire a small syntax extension to aggregate functionscalled in a SELECT statement. Alternatively, horizontalaggregations can be used to generate SQL code from a datamining tool to build data sets for data mining analysis. Westart by explaining how to automatically generate SQL code.

3.1 SQL Code Generation

Our main goal is to define a template to generate SQL codecombining aggregation and transposition (pivoting). Asecond goal is to extend the SELECT statement with aclause that combines transposition with aggregation. Con-sider the following GROUP BY query in standard SQL thattakes a subset L1; . . . ; Lm from D1; . . . ; Dp:

SELECT L1; ::; Lm, sum(A)

FROM F

GROUP BY L1; . . . ; Lm;

This aggregation query will produce a wide table withmþ 1 columns (automatically determined), with one groupfor each unique combination of values L1; . . . ; Lm and oneaggregated value per group (sum(A) in this case). In order toevaluate this query the query optimizer takes three inputparameters: 1) the input table F , 2) the list of groupingcolumnsL1; . . . ; Lm, 3) the column to aggregate (A). The basicgoal of a horizontal aggregation is to transpose (pivot) theaggregated column A by a column subset of L1; . . . ; Lm; forsimplicity assume such subset is R1; . . . ; Rk where k < m. Inother words, we partition the GROUP BY list into twosublists: one list to produce each group (j columnsL1; . . . ; Lj)and another list (k columns R1; . . . ; Rk) to transposeaggregated values, where fL1; . . . ; Ljg \ fR1; . . . ; Rkg ¼ ;.Each distinct combination of fR1; . . . ; Rkgwill automaticallyproduce an output column. In particular, if k ¼ 1 then thereare j�R1

ðF Þj columns (i.e., each value inR1 becomes a columnstoring one aggregation). Therefore, in a horizontal aggrega-tion there are four input parameters to generate SQL code:

1. the input table F ,2. the list of GROUP BY columns L1; . . . ; Lj,3. the column to aggregate (A),4. the list of transposing columns R1; . . . ; Rk.

Horizontal aggregations preserve evaluation semantics ofstandard (vertical) SQL aggregations. The main differencewill be returning a table with a horizontal layout, possiblyhaving extra nulls. The SQL code generation aspect is



explained in technical detail in Section 3.4. Our definitionallows a straightforward generalization to transpose multi-ple aggregated columns, each one with a different list oftransposing columns.

3.2 Proposed Syntax in Extended SQL

We now turn our attention to a small syntax extension tothe SELECT statement, which allows understanding ourproposal in an intuitive manner. We must point out theproposed extension represents nonstandard SQL becausethe columns in the output table are not known whenthe query is parsed. We assume F does not change while ahorizontal aggregation is evaluated because new valuesmay create new result columns. Conceptually, we extendstandard SQL aggregate functions with a “transposing” BYclause followed by a list of columns (i.e., R1; . . . ; Rk), toproduce a horizontal set of numbers instead of one number.Our proposed syntax is as follows:

SELECT L1; ::; Lj, HðABYR1; . . . ; RkÞFROM F

GROUP BY L1; . . . ; Lj;

We believe the subgroup columns R1; . . . ; Rk should be aparameter associated to the aggregation itself. That is whythey appear inside the parenthesis as arguments, butalternative syntax definitions are feasible. In the context ofour work, HðÞ represents some SQL aggregation (e.g.,sumðÞ, countðÞ, minðÞ, maxðÞ, avgðÞ). The function HðÞ musthave at least one argument represented by A, followed by alist of columns. The result rows are determined by columnsL1; . . . ; Lj in the GROUP BY clause if present. Resultcolumns are determined by all potential combinations ofcolumns R1; . . . ; Rk, where k ¼ 1 is the default. Also,fL1; . . . ; Ljg \ fR1; . . . ; Rkg ¼ ;

We intend to preserve standard SQL evaluation seman-tics as much as possible. Our goal is to develop sound andefficient evaluation mechanisms. Thus, we propose thefollowing rules.

1. The GROUP BY clause is optional, like a verticalaggregation. That is, the list L1; . . . ; Lj may be empty.When the GROUP BY clause is not present then thereis only one result row. Equivalently, rows can begrouped by a constant value (e.g., L1 ¼ 0) to alwaysinclude a GROUP BY clause in code generation.

2. When the clause GROUP BY is present there shouldnot be a HAVING clause that may produce cross-tabulation of the same group (i.e., multiple rowswith aggregated values per group).

3. The transposing BY clause is optional. When BY isnot present then a horizontal aggregation reduces toa vertical aggregation.

4. When the BY clause is present the list R1; . . . ; Rk isrequired, where k ¼ 1 is the default.

5. Horizontal aggregations can be combined withvertical aggregations or other horizontal aggrega-tions on the same query, provided all use the sameGROUP BY columns fL1; . . . ; Ljg.

6. As long as F does not change during queryprocessing horizontal aggregations can be freelycombined. Such restriction requires locking [11],which we will explain later.

7. The argument to aggregate represented by A isrequired; A can be a column name or an arithmeticexpression. In the particular case of countðÞA can bethe “DISTINCT” keyword followed by the list ofcolumns.

8. When HðÞ is used more than once, in different terms,it should be used with different sets of BY columns.

3.2.1 Examples

In a data mining project, most of the effort is spent inpreparing and cleaning a data set. A big part of this effortinvolves deriving metrics and coding categorical attributesfrom the data set in question and storing them in a tabular(observation, record) form for analysis so that they can beused by a data mining algorithm.

Assume we want to summarize sales information withone store per row for one year of sales. In more detail, weneed the sales amount broken down by day of the week, thenumber of transactions by store per month, the number ofitems sold by department and total sales. The followingquery in our extended SELECT syntax provides the desireddata set, by calling three horizontal aggregations.

SELECT

storeId,

sum(salesAmt BY dayofweekName),

count(distinct transactionid BY salesMonth),

sum(1 BY deptName),

sum(salesAmt)

FROM transactionLine,DimDayOfWeek,DimDepartment,DimMonth

WHERE salesYear¼2009

AND transactionLine.dayOfWeekNo

¼DimDayOfWeek.dayOfWeekNo

AND transactionLine.deptId

¼DimDepartment.deptId

AND transactionLine.MonthId

¼DimTime.MonthIdGROUP BY storeId;

This query produces a result table like the one shown inTable 1. Observe each horizontal aggregation effectivelyreturns a set of columns as result and there is call to astandard vertical aggregation with no subgrouping col-umns. For the first horizontal aggregation, we show daynames and for the second one we show the number of dayof the week. These columns can be used for linearregression, clustering, or factor analysis. We can analyzecorrelation of sales based on daily sales. Total sales can bepredicted based on volume of items sold each day of theweek. Stores can be clustered based on similar sales for eachday of the week or similar sales in the same department.

Consider a more complex example where we want toknow for each store subdepartment how sales compare foreach region-month showing total sales for each region/month combination. Subdepartments can be clusteredbased on similar sales amounts for each region/monthcombination. We assume all stores in all regions have thesame departments, but local preferences lead to differentbuying patterns. This query in our extended SELECT buildsthe required data set:



SELECT subdeptid,sum(salesAmt BY regionNo,monthNo)


GROUP BY subdeptId;

We turn our attention to another common data prepara-tion task, transforming columns with categorical attributesinto binary columns. The basic idea is to create a binarydimension for each distinct value of a categorical attribute.This can be accomplished by simply calling maxð1BY::Þ,grouping by the appropriate columns. The next queryproduces a vector showing 1 for the departments where thecustomer made a purchase, and 0 otherwise.

SELECT

transactionId,

max(1 BY deptId DEFAULT 0)


GROUP BY transactionId;

3.3 SQL Code Generation: Locking and TableDefinition

In this section, we discuss how to automatically generateefficient SQL code to evaluate horizontal aggregations.Modifying the internal data structures and mechanisms ofthe query optimizer is outside the scope of this paper, butwe give some pointers. We start by discussing the structureof the result table and then query optimization methods topopulate it. We will prove the three proposed evaluationmethods produce the same result table FH .

3.3.1 Locking

In order to get a consistent query evaluation it is necessaryto use locking [7], [11]. The main reasons are that anyinsertion into F during evaluation may cause inconsisten-cies: 1) it can create extra columns in FH , for a newcombination of R1; . . . ; Rk; 2) it may change the number ofrows of FH , for a new combination of L1; . . . ; Lj; 3) it maychange actual aggregation values in FH . In order to returnconsistent answers, we basically use table-level locks on F ,FV , and FH acquired before the first statement starts andreleased after FH has been populated. In other words, theentire set of SQL statements becomes a long transaction. Weuse the highest SQL isolation level: SERIALIZABLE. Noticean alternative simpler solution would be to use a static(read-only) copy of F during query evaluation. That is,horizontal aggregations can operate on a read-only data-base without consistency issues.

3.3.2 Result Table Definition

Let the result table be FH . Recall from Section 2 FH has daggregation columns, plus its primary key. The horizontalaggregation function HðÞ returns not a single value, but a

set of values for each group L1; . . . ; Lj. Therefore, the resulttable FH must have as primary key the set of groupingcolumns fL1; . . . ; Ljg and as nonkey columns all existingcombinations of values R1; . . . ; Rk. We get the distinct valuecombinations of R1; . . . ; Rk using the following statement.

SELECT DISTINCT R1; ::; Rk

FROM F ;

Assume this statement returns a table with ddistinct rows.Then each row is used to define one column to store anaggregation for one specific combination of dimensionvalues. Table FH that has fL1; . . . ; Ljg as primary key and dcolumns corresponding to each distinct subgroup. Therefore,FH has d columns for data mining analysis and jþ d columnsin total, where each Xj corresponds to one aggregated valuebased on a specific R1; . . . ; Rk values combination.

CREATE TABLE FH(

L1 int

,. . .

,Lj int

,X1 real

,. . .

,Xd real

) PRIMARY KEY(L1; . . . ; Lj);

3.4 SQL Code Generation: Query EvaluationMethods

We propose three methods to evaluate horizontal aggrega-tions. The first method relies only on relational operations.That is, only doing select, project, join, and aggregationqueries; we call it the SPJ method. The second form relies onthe SQL “case” construct; we call it the CASE method. Eachtable has an index on its primary key for efficient joinprocessing. We do not consider additional indexingmechanisms to accelerate query evaluation. The thirdmethod uses the built-in PIVOT operator, which transformsrows to columns (e.g., transposing). Figs. 2 and 3 show anoverview of the main steps to be explained below (for asum() aggregation).

3.4.1 SPJ Method

The SPJ method is interesting from a theoretical point ofview because it is based on relational operators only. Thebasic idea is to create one table with a vertical aggregationfor each result column, and then join all those tables toproduce FH . We aggregate from F into d projected tableswith d Select-Project-Join-Aggregation queries (selection,projection, join, aggregation). Each table FI corresponds toone subgrouping combination and has fL1; . . . ; Ljg asprimary key and an aggregation on A as the only nonkey


TABLE 1A Multidimensional Data Set in Horizontal Layout, Suitable for Data Mining


column. It is necessary to introduce an additional table F0,that will be outer joined with projected tables to get acomplete result set. We propose two basic substrategies tocompute FH . The first one directly aggregates from F . Thesecond one computes the equivalent vertical aggregation ina temporary table FV grouping by L1; . . . ; Lj; R1; . . . ; Rk.Then horizontal aggregations can be instead computedfrom FV , which is a compressed version of F , sincestandard aggregations are distributive [9].

We now introduce the indirect aggregation based on theintermediate table FV , that will be used for both the SPJ andthe CASE method. Let FV be a table containing the verticalaggregation, based on L1; . . . ; Lj; R1; . . . ; Rk. Let V() repre-sent the corresponding vertical aggregation for HðÞ. Thestatement to compute FV gets a cube:

INSERT INTO FVSELECT L1; . . . ; Lj; R1; . . . ; Rk, V(A)FROM F

GROUP BY L1; . . . ; Lj; R1; . . . ; Rk;

Table F0 defines the number of result rows, and buildsthe primary key. F0 is populated so that it contains everyexisting combination of L1; . . . ; Lj. Table F0 has fL1; . . . ; Ljgas primary key and it does not have any nonkey column.

INSERT INTO F0

SELECT DISTINCT L1; . . . ; LjFROM fF jFV g;

In the following discussion I 2 f1; . . . ; dg: we use h to

make writing clear, mainly to define boolean expressions.We need to get all distinct combinations of subgroupingcolumns R1; . . . ; Rk, to create the name of dimension

columns, to get d, the number of dimensions, and togenerate the boolean expressions for WHERE clauses. EachWHERE clause consists of a conjunction of k equalities

based on R1; . . . ; Rk.

SELECT DISTINCT R1; . . . ; Rk

FROM fF jFV g;Tables F1; . . . ; Fd contain individual aggregations for

each combination of R1; . . . ; Rk. The primary key of table FIis fL1; . . . ; Ljg.

INSERT INTO FISELECT L1; . . . ; Lj; V ðAÞFROM fF jFV gWHERE R1 ¼ v1I AND .. AND Rk ¼ vkIGROUP BY L1; . . . ; Lj;

Then each table FI aggregates only those rows that

correspond to the Ith unique combination of R1; . . . ; Rk,

given by the WHERE clause. A possible optimization is

synchronizing table scans to compute the d tables in one pass.Finally, to get FH we need d left outer joins with the dþ 1

tables so that all individual aggregations are properly

assembled as a set of d dimensions for each group. Outer

joins set result columns to null for missing combinations for

the given group. In general, nulls should be the default

value for groups with missing combinations. We believe it

would be incorrect to set the result to zero or some other

number by default if there are no qualifying rows. Such

approach should be considered on a per-case basis.

INSERT INTO FHSELECT

F0:L1; F0:L2; . . . ; F0:Lj,

F1:A; F2:A; . . . ; Fd:A

FROM F0

LEFT OUTER JOIN F1

ON F0:L1 ¼ F1:L1 and . . . and F0:Lj ¼ F1:LjLEFT OUTER JOIN F2

ON F0:L1 ¼ F2:L1 and . . . and F0:Lj ¼ F2:Lj. . .

LEFT OUTER JOIN FdON F0:L1 ¼ Fd:L1 and . . . and F0:Lj ¼ Fd:Lj;

This statement may look complex, but it is easy to see that

each left outer join is based on the same columns L1; . . . ; Lj.

To avoid ambiguity in column references, L1; . . . ; Lj are

qualified with F0. Result column I is qualified with table FI .

Since F0 has n rows each left outer join produces a partial

table withn rows and one additional column. Then at the end,

FH will have n rows and d aggregation columns. The

statement above is equivalent to an update-based strategy.

Table FH can be initialized inserting n rows with key

L1; . . . ; Lj and nulls on the ddimension aggregation columns.

Then FH is iteratively updated from FI joining on L1; . . . ; Lj.

This strategy basically incurs twice I/O doing updates

instead of insertion. Reordering the d projected tables to join

cannot accelerate processing because each partial table has n

rows. Another claim is that it is not possible to correctly

compute horizontal aggregations without using outer joins.

In other words, natural joins would produce an incomplete

result set.


Fig. 2. Main steps of methods based on F (unoptimized).

Fig. 3. Main steps of methods based on FV (optimized).


3.4.2 CASE Method

For this method, we use the “case” programming constructavailable in SQL. The case statement returns a valueselected from a set of values based on boolean expressions.From a relational database theory point of view this isequivalent to doing a simple projection/aggregation querywhere each nonkey value is given by a function that returnsa number based on some conjunction of conditions. Wepropose two basic substrategies to compute FH . In a similarmanner to SPJ, the first one directly aggregates from F andthe second one computes the vertical aggregation in atemporary table FV and then horizontal aggregations areindirectly computed from FV .

We now present the direct aggregation method. Hor-izontal aggregation queries can be evaluated by directlyaggregating from F and transposing rows at the same timeto produce FH . First, we need to get the unique combina-tions of R1; . . . ; Rk that define the matching booleanexpression for result columns. The SQL code to computehorizontal aggregations directly from F is as follows:observe V ðÞ is a standard (vertical) SQL aggregation thathas a “case” statement as argument. Horizontal aggrega-tions need to set the result to null when there are noqualifying rows for the specific horizontal group to beconsistent with the SPJ method and also with the extendedrelational model [4].


FROM F ;

INSERT INTO FHSELECT L1; . . . ; Lj

,V(CASE WHEN R1 ¼ v11 and . . . and Rk ¼ vk1

THEN A ELSE null END)

..

,V(CASE WHEN R1 ¼ v1d and . . . and Rk ¼ vkdTHEN A ELSE null END)

FROM F

GROUP BY L1; L2; . . . ; Lj;

This statement computes aggregations in only one scanon F . The main difficulty is that there must be a feedbackprocess to produce the “case” boolean expressions.

We now consider an optimized version using FV . Basedon FV , we need to transpose rows to get groups based onL1; . . . ; Lj. Query evaluation needs to combine the desiredaggregation with “CASE” statements for each distinctcombination of values of R1; . . . ; Rk. As explained above,horizontal aggregations must set the result to null whenthere are no qualifying rows for the specific horizontalgroup. The boolean expression for each case statement has aconjunction of k equality comparisons. The followingstatements compute FH :


FROM FV ;

INSERT INTO FHSELECT L1,..,Lj

,sum(CASE WHEN R1 ¼ v11 and .. and Rk ¼ vk1

THEN A ELSE null END)

..

,sum(CASE WHEN R1 ¼ v1d and .. and Rk ¼ vkdTHEN A ELSE null END)

FROM FVGROUP BY L1; L2; . . . ; Lj;

As can be seen, the code is similar to the code presented

before, the main difference being that we have a call to

sumðÞ in each term, which preserves whatever values were

previously computed by the vertical aggregation. It has the

disadvantage of using two tables instead of one as requiredby the direct computation from F . For very large tables F

computing FV first, may be more efficient than computing

directly from F .

3.4.3 PIVOT Method

We consider the PIVOT operator which is a built-in operator

in a commercial DBMS. Since this operator can perform

transposition it can help evaluating horizontal aggregations.

The PIVOT method internally needs to determine how many

columns are needed to store the transposed table and it can

be combined with the GROUP BY clause.The basic syntax to exploit the PIVOT operator to

compute a horizontal aggregation assuming one BY column

for the right key columns (i.e., k ¼ 1) is as follows:

SELECT DISTINCT R1

FROM F ; /* produces v1; . . . ; vd */

SELECT L1; L2; . . . ; Lj,v1; v2; . . . ; vd

INTO FtFROM F

PIVOT(

V(A) FOR R1 in (v1; v2; . . . ; vd)) AS P;

SELECT

L1; L2; . . . ; Lj,V ðv1Þ; V ðv2Þ; . . . ; V ðvdÞ

INTO FHFROM FtGROUP BY L1; L2; . . . ; Lj;

This set of queries may be inefficient because Ft can be a

large intermediate table. We introduce the following

optimized set of queries which reduces of the intermediate

table:

SELECT DISTINCT R1

FROM F ; /* produces v1; . . . ; vd */

SELECT

L1; L2; . . . ; Lj,v1; v2; . . . ; vd

INTO FHFROM (

SELECT L1; L2; . . . ; Lj; R1; A

FROM F ) FtPIVOT(

V ðAÞ FOR R1 in (v1; v2; . . . ; vd)

) AS P;



Notice that in the optimized query the nested query

trims F from columns that are not later needed. That is,

the nested query projects only those columns that will

participate in FH . Also, the first and second queries can be

computed from FV ; this optimization is evaluated in

Section 4.

3.4.4 Example of Generated SQL Queries

We now show actual SQL code for our small example. This

SQL code produces FH in Fig. 1. Notice the three methods

can compute from either F or FV , but we use F to make

code more compact.The SPJ method code is as follows (computed from F ):

/* SPJ method */

INSERT INTO F1

SELECT D1,sum(A) AS A

FROM F

WHERE D2=’X’

GROUP BY D1;

INSERT INTO F2

SELECT D1,sum(A) AS A

FROM F

WHERE D2=’Y’

GROUP BY D1;

INSERT INTO FH

SELECT F0.D1,F1.A AS D2_X,F2.A AS D2_Y

FROM F0 LEFT OUTER JOIN F1 on F0.D1=F1.D1

LEFT OUTER JOIN F2 on F0.D1=F2.D1;

The CASE method code is as follows (computed from F ):

/* CASE method */

INSERT INTO FH

SELECT

D1

,SUM(CASE WHEN D2=’X’ THEN A

ELSE null END) as D2_X

,SUM(CASE WHEN D2=’Y’ THEN A

ELSE null END) as D2_Y

FROM F

GROUP BY D1;

Finally, the PIVOT method SQL is as follows (computed

from F ):

/* PIVOT method */

INSERT INTO FH

SELECT

D1

,[X] as D2_X

,[Y] as D2_Y

FROM (

SELECT D1, D2, A FROM F

) as p

PIVOT (

SUM(A)

FOR D2 IN ([X], [Y])

) as pvt;

3.5 Properties of Horizontal Aggregations

A horizontal aggregation exhibits the following properties:

1. n ¼ jFH j matches the number of rows in a verticalaggregation grouped by L1; . . . ; Lj.

2. d ¼ j�R1;...;RkðF Þj.

3. Table FH may potentially store more aggregatedvalues than FV due to nulls. That is, jFV j � nd.

3.6 Equivalence of Methods

We will now prove the three methods produce the sameresult.

Theorem 1. SPJ and CASE evaluation methods produce the same

result.

Proof. Let S ¼ �R1¼v1I\��\Rk¼vkI ðF Þ. Each table FI in SPJ iscomputed as FI ¼ L1;...;LjF V ðAÞðSÞ. The F notation is usedto extend relational algebra with aggregations: theGROUP BY columns are L1 . . .Lj and the aggregationfunction is V ðÞ. Note: in the following equations all joinsffl are left outer joins. We can follow an induction on d,the number of distinct combinations for R1; . . . ; Rk.When d ¼ 1 (base case) it holds j�R1...Rk

ðF Þj ¼ 1 andS1 ¼ �R1¼v11\...Rk¼vk1

ðF Þ. Then F1 ¼ L1;...;LjF V ðAÞðS1Þ. Bydefinition F0 ¼ �L1;...;LjðF Þ. Since

j�R1...RkðF Þj ¼ 1j�L1;...;LjðF Þj ¼ j�L1;...;Lj;R1;...;Rk

ðF Þj:

Then FH ¼ F0 ffl F1 ¼ F1 (the left join does not insertnulls). On the other hand, for the CASE method letG ¼ L1;...;LjF V ð�ðA;1ÞÞðF Þ, where �ð; IÞ represents the CASEstatement and I is the Ith dimension. But sincej�R1...Rk

ðF Þj ¼ 1 then

G ¼ L1;...;LjF V ð�ðA;1ÞÞðF Þ ¼ L1;...;LjF V ðAÞðF Þ

(i.e., the conjunction in �ðÞ always evaluates to true).Therefore, G ¼ F1, which proves both methods returnthe same result. For the general case, assume the resultholds for d� 1. Consider F0 ffl F1 . . . ffl Fd. By theinduction hypothesis this means

F0 ffl F1 . . . ffl Fd�1 ¼ L1;...;LjF V ð�ðA;1ÞÞ;V ð�ðA;2ÞÞ;...;V ð�ðA;d�1ÞÞðF Þ:

Let us analyze Fd. Table Sd ¼ �R1¼v1d\...Rk¼vkdðF Þ. TableFd ¼ L1;...;LjFV ðAÞðSdÞ. Now, F0 ffl Fd augments Fd withnulls so that jF0j ¼ jFH j. Since the dth conjunction is the

same for Fd and for �ðA; dÞ. Then

F0 ffl Fd ¼ L1;...;LjF V ð�ðA;dÞÞðF Þ:

Finally,

F0 ffl F1 ffl . . .Fd

¼ L1;...;LjF V ð�ðA;1ÞÞ;V ð�ðA;2ÞÞ;...;V ð�ðA;d�1ÞÞ;V ð�ðA;dÞÞðF Þ:ut

Theorem 2. The CASE and PIVOT evaluation methods produce

the same result.

Proof. (sketch) The SQL PIVOT operator works in a similarmanner to the CASE method. We consider the optimizedversion of PIVOT, where we project only the columns



required by FH (i.e., trimming F ). When the PIVOToperator is applied one aggregation column is producedfor every distinct value vj, producing the d desiredcolumns. An induction on j proves the clause “V ðAÞ forR1 IN (v1; . . . ; vd)” is transformed into d “V(CASE WHENR1 ¼ vj THEN A END)” statements, where each pivotedvalue produces one Xj for j ¼ 1 . . . d. Notice a GROUPBY clause is not required for the outer SELECT statementfor the optimized PIVOT version. tu

Based on the two previous theoretical results we presentour main theorem.

Theorem 3. Given the same input table F and horizontalaggregation query, the SPJ, CASE, and PIVOT methodsproduce the same result.

3.7 Time Complexity and I/O Cost

We now analyze time complexity for each method. Recallthat N ¼ jF j, n ¼ jFH j and d is the data set dimensionality(number of cross-tabulated aggregations). We consider oneI/O to read/write one row as the basic unit to analyze thecost to evaluate the query. This analysis considers everymethod precomputes FV .

SPJ: We assume hash or sort-merge joins [7] are available.Thus a join between two tables of size OðnÞ can be evaluatedin time OðnÞ on average. Otherwise, joins take timeOðnlog2nÞ. Computing the sort in the initial query “SELECTDISTINCT. . . ” takes OðNlog2ðNÞÞ. If the right key producesa high d (say d � 10 and a uniform distribution of values).Then each � query will have a high selectivity predicate. EachjFij � n. Therefore, we can expect jFij < N . There ared� queries with different selectivity with a conjunction ofk terms OðknþNÞ each. Then total time for all selectionqueries is Oðdknþ dNÞ. There are d GROUP-BY operationswith L1; . . . ; Lj producing a table OðnÞ each. Therefore, thedGROUP-BYs take timeOðdnÞwith I/O cost 2dn (to read andwrite). Finally, there are d outer joins taking OðnÞ orOðnlog2ðnÞÞ each, giving a total time OðdnÞ or Oðdnlog2ðnÞÞ.In short, time is OðN log2ðNÞ þ dknþ dNÞ and I/O cost isN log2ðNÞ þ 3dnþ dN with hash joins. Otherwise, time isOðNlog2ðNÞ þ dknlog2ðnÞ þ dNÞ and I/O cost is N log2ðNÞ þ2dnþ dnlog2ðnÞ þ dN with sort-merge joins.

Time depends on number of distinct values, theircombination, and probabilistic distribution of values.

CASE: Computing the sort in the initial query “SELECTDISTINCT. . . ” takes OðN log2ðNÞÞ. There are OðdkNÞcomparisons; notice this is fixed. There is one GROUP-BYwith L1; . . . ; Lj in time OðdknÞ producing table OðdnÞ.Evaluation time depends on the number of distinct valuecombinations, but not on their probabilistic distribution. Inshort, time is OðNlog2ðNÞ þ dknþNÞ and I/O cost isN log2ðNÞ þ nþN . As we can see, time complexity is thesame, but I/O cost is significantly smaller compared to SPJ.

PIVOT: We consider the optimized version which trimsF from irrelevant columns and k ¼ 1. Like the SPJ andCASE methods, PIVOT depends on selecting the distinctvalues from the right keys R1; . . . ; Rk. It avoids joins andsaves I/O when it receives as input the trimmed version ofF . Then it has similar time complexity to CASE. Also, timedepends on number of distinct values, their combination,and probabilistic distribution of values.

3.8 Discussion

For all proposed methods to evaluate horizontal aggrega-tions we summarize common requirements.

1. All methods require grouping rows by L1; . . . ; Lj inone or several queries.

2. All methods must initially get all distinct combina-tions of R1; . . . ; Rk to know the number and names ofresult columns. Each combination will match aninput row with a result column. This step makesquery optimization difficult by standard queryoptimization methods because such columns cannotbe known when a horizontal aggregation query isparsed and optimized.

3. It is necessary to set result columns to null whenthere are no qualifying rows. This is done either byouter joins or by the CASE statement.

4. Computation can be accelerated in some cases byfirst computing FV and then computing furtheraggregations from FV instead of F . The amount ofacceleration depends on how larger is N withrespect to n (i.e., if N � n). These requirementscan be used to develop more efficient queryevaluation algorithms.

The correct way to treat missing combinations for onegroup is to set the result column to null. But in some cases itmay make sense to change nulls to zero, as was the case tocode a categorical attribute into binary dimensions. Someaspects about both CASE substrategies are worth discussingin more depth. Notice the boolean expressions in each termproduce disjoint subsets (i.e., they partition F ). The queriesabove can be significantly accelerated using a smarterevaluation because each input row falls on only one resultcolumn and the rest remain unaffected. Unfortunately, theSQL parser does not know this fact and it unnecessarilyevaluates d boolean expressions for each input row in F .This requires OðdÞ time complexity for each row, instead ofOð1Þ. In theory, the SQL query optimizer could reduce thenumber to conjunctions to evaluate down to one using ahash table that maps one conjunction to one dimensioncolumn. Then the complexity for one row can decrease fromOðdÞ down to Oð1Þ.

If an input query has several terms having a mix ofhorizontal aggregations and some of them share similarsubgrouping columns R1; . . . ; Rk the query optimizer canavoid redundant comparisons by reordering operations. If apair of horizontal aggregations does not share the same set ofsubgrouping columns further optimization is not possible.

Horizontal aggregations should not be used when the setof columns fR1; . . . ; Rkg have many distinct values (such asthe primary key of F ). For instance, getting horizontalaggregations on transactionLine using itemId. In theorysuch query would produce a very wide and sparse table,but in practice it would cause a runtime error because themaximum number of columns allowed in the DBMS couldbe exceeded.

3.9 DBMS Limitations

There exist two DBMS limitations with horizontal aggrega-tions: reaching the maximum number of columns in onetable and reaching the maximum column name lengthwhen columns are automatically named. To elaborate onthis, a horizontal aggregation can return a table that goes



beyond the maximum number of columns in the DBMSwhen the set of columns fR1; . . . ; Rkg has a large number ofdistinct combinations of values, or when there are multiplehorizontal aggregations in the same query. On the otherhand, the second important issue is automatically generat-ing unique column names. If there are many subgroupingcolumns R1; . . . ; Rk or columns are of string data types, thismay lead to generate very long column names, which mayexceed DBMS limits. However, these are not importantlimitations because if there are many dimensions that islikely to correspond to a sparse matrix (having many zeroesor nulls) on which it will be difficult or impossible tocompute a data mining model. On the other hand, the largecolumn name length can be solved as explained below.

The problem of d going beyond the maximum number ofcolumns can be solved by vertically partitioning FH so thateach partition table does not exceed the maximum numberof columns allowed by the DBMS. Evidently, each partitiontable must have L1; . . . ; Lj as its primary key. Alternatively,the column name length issue can be solved by generatingcolumn identifiers with integers and creating a “dimension”description table that maps identifiers to full descriptions,but the meaning of each dimension is lost. An alternative isthe use of abbreviations, which may require manual input.

4 EXPERIMENTAL EVALUATION

In this section, we present our experimental evaluation on acommercial DBMS. We evaluate query optimizations,compare the three query evaluation methods, and analyzetime complexity varying table sizes and output data setdimensionality.

4.1 Setup: Computer Configuration and Data Sets

We used SQL Server V9, running on a DBMS serverrunning at 3.2 GHz, Dual Core processor, 4 GB of RAM and1 TB on disk. The SQL code generator was programmed inthe Java language and connected to the server via JDBC.The PIVOT operator was used as available in the SQLlanguage implementation provided by the DBMS.

We used large synthetic data sets described below. Weanalyzed queries having only one horizontal aggregation,with different grouping and horizontalization columns.Each experiment was repeated five times and we report theaverage time in seconds. We cleared cache memory beforeeach method started in order to evaluate query optimiza-tion under pessimistic conditions.

We evaluated optimization strategies for aggregationqueries with synthetic data sets generated by the TPC-H

generator. In general, we evaluated horizontal aggregationqueries using the fact table transactionLine as input. Table 2shows the specific columns from the TPC-H fact table weuse for the left key and the right key in a horizontalaggregation. Basically, we picked several column combina-tions to get different d and n. In order to get meaningfuldata sets for data mining we picked high selectivitycolumns for the left key and low selectivity columns forthe right key. In this manner d� n, which is the mostcommon scenario in data mining.

Since TPC-H only creates uniformly distributed valueswe created a similar data generator of our own to controlthe probabilistic distribution of column values. We alsocreated synthetic data sets controlling the number ofdistinct values in grouping keys and their probabilisticdistribution. We used two probabilistic distributions: uni-form (unskewed) and zipf (skewed), which represent twocommon and complementary density functions in queryoptimization. The table definition for these data sets issimilar to the fact table from TPC-H. The key valuedistribution has an impact both in grouping rows andwriting rows to the output table.

4.2 Query Optimizations

Table 3 analyzes our first query optimization, applied tothree methods. Our goal is to assess the accelerationobtained by precomputing a cube and storing it on FV .We can see this optimization uniformly accelerates allmethods. This optimization provides a different gain,depending on the method: for SPJ the optimization is bestfor small n, for PIVOT for large n and for CASE there israther a less dramatic improvement all across n. It isnoteworthy PIVOT is accelerated by our optimization,despite the fact it is handled by the query optimizer. Sincethis optimization produces significant acceleration for thethree methods (at least 2 faster) we will use it by default.Notice that precomputing FV takes the same time withineach method. Therefore, comparisons are fair.

We now evaluate an optimization specific to the PIVOToperator. This PIVOT optimization is well known, as welearned from SQL Server DBMS users groups. Table 4shows the impact of removing (trimming) columns notneeded by PIVOT. That is, removing columns that will notappear in FH . We can see the impact is significant,accelerating evaluation time from three to five times. Allour experiments incorporate this optimization by default.

4.3 Comparing Evaluation Methods

Table 5 compares the three query optimization methods.Notice Table 5 is a summarized version of Table 3, showing


TABLE 2Summary of Grouping Columns from

TPC-H Table Transactionline(N ¼ 6M)

TABLE 3Query Optimization: Precompute Vertical Aggregation

in FV (N ¼ 12M). Times in Seconds


best times for each method. Table 6 is a complement,showing time variability around the mean time � for timesreported in Table 5; we show one standard deviation � andpercentage that one � represents respect to �. As can beseen, times exhibit small variability, PIVOT exhibitssmallest variability, followed by CASE. As we explainedbefore, in the time complexity and I/O cost analysis, thetwo main factors influencing query evaluation time are dataset size and grouping columns (dimensions) cardinalities.We consider different combinations of L1; . . . ; Lj andR1; . . . ; Rk columns to get different values of n and d,respectively. Refer to Table 2 to know the correspondencebetween TPC-H columns and FH table sizes d; n. The threemethods use the optimization precomputing FV in order tomake a uniform and fair comparison (recall this optimiza-tion works well for large tables). In general, SPJ is theslowest method, as expected. On the other hand, PIVOTand CASE have similar time performance with slightlybigger differences in some cases. Time grows as n grows forall methods, but much more for SPJ. On the other hand,there is a small time growth when d increases for SPJ, butnot a clear trend for PIVOT and CASE. Notice we arecomparing against the optimized version of the PIVOTmethod (the CASE method is significantly faster than theunoptimized PIVOT method). We conclude that n is themain performance factor for PIVOT and CASE methods,whereas both d and n impact the SPJ method.

We investigated the reason for time differences analyz-ing query evaluation plans. It turned out PIVOT and CASEhave similar query evaluation plans when they use FV , butthey are slightly different when they use F . The mostimportant difference is that the PIVOT and CASE methodshave a parallel step which allows evaluating aggregationsin parallel. Such parallel step does not appear when PIVOTevaluates the query from F . Therefore, in order to conduct afair comparison we use FV by default.

4.4 Time Complexity

We now verify the time complexity analysis given inSection 3.7. We plot time complexity keeping varying oneparameter and the remaining parameters fixed. In theseexperiments, we generated synthetic data sets similar tothe fact table of TPC-H of different sizes with groupingcolumns of varying selectivities (number of distinct values).We consider two basic probabilistic distribution of values:uniform (unskewed) and zipf (skewed). The uniformdistribution is the distribution used by default.

Fig. 4 shows the impact of increasing the size of the facttable (N). The left plot analyzes a small FH table (n ¼ 1k),whereas the right plot presents a much larger FH table(n ¼ 128k). Recall from Section 3.7 time is OðNlog2NÞ þNwhen N grows and the other sizes (n; d) are fixed (the firstterm corresponds to SELECT DISTINCT and the second oneto the method). The trend indicates all evaluation methodsget indeed impacted by N . Times tend to be linear as Ngrows for the three methods, which shows the methodqueries have more weight than getting the distinct dimen-sion combinations. The PIVOT and CASE methods showvery similar performance. On the other hand, there is a biggap between PIVOT/CASE and SPJ for large n. The trendindicates such gap will not shrink as N grows.

Fig. 5 now leaves N fixed and computes horizontalaggregations increasing n. The left plot uses d ¼ 16 (to getthe individual horizontal aggregation columns) and theright plot shows d ¼ 64. Recall from Section 3.7, that timeshould grow OðnÞ keeping N and d fixed. Clearly time islinear and grows slowly for PIVOT and CASE. On the otherhand, time grows faster for SPJ and the trend indicates it ismostly linear, especially for d ¼ 64. From a comparisonperspective, PIVOT and CASE have similar performance forlow d and the same performance for high d. SPJ is muchslower, as expected. In short, for PIVOT and CASE timecomplexity is OðnÞ, keeping N; d fixed.

Fig. 6 shows time complexity varying d. According to ouranalysis in Section 3.7, time should be OðdÞ. The left plotshows a medium sized data set with n ¼ 32 k, whereas theright plot shows a large data set with n ¼ 128 k. For small n(left plot) the trend for PIVOT and CASE methods is notclear; overall time is almost constant overall. On the otherhand, for SPJ time indeed grows in a linear fashion. For largen (right plot) trends are clean; PIVOT and CASE methodshave the same performance, its time is clearly linear,although with a small slope. On the other hand, time is notlinear overall for SPJ; it initially grows linearly, but it has ajump beyond d ¼ 64. The reason is the size of the inter-mediate join result tables, which get increasingly wider.


TABLE 4Query Optimization: Remove(Trim) Unnecessary Columns

from FV for PIVOT (N ¼ 12M).Times in Seconds

TABLE 5Comparing Query Evaluation Methods(All with Optimization Computing FV ).

Times in Seconds

TABLE 6Variability of Mean Time (N ¼ 12M, One Standard

Deviation, Percentage of Mean Time).Times in Seconds


Table 7 analyzes the impact of having an uniform or azipf distribution of values on the right key. That is, weanalyze the impact of an unskewed and a skeweddistribution. In general, all methods are impacted by thedistribution of values, but the impact is significant when dis high. Such trend indicates higher I/O cost to updateaggregations on more subgroups. The impact is marginalfor all methods at low d. PIVOT shows a slightly higherimpact than the CASE method by the skewed distributionat high d. But overall, both show similar behavior. SPJ isagain the slowest and shows bigger impact at high d.

5 RELATED WORK

We first discuss research on extending SQL code for datamining processing. We briefly discuss related work onquery optimization. We then compare horizontal aggrega-tions with alternative proposals to perform transpositionor pivoting.

There exist many proposals that have extended SQLsyntax. The closest data mining problem associated toOLAP processing is association rule mining [18]. SQLextensions to define aggregate functions for association rule


Fig. 4. Time complexity varying N (d ¼ 16, uniform distribution).

Fig. 5. Time complexity varying n (N ¼ 4M, uniform distribution).

Fig. 6. Time complexity varying d (N ¼ 4M, uniform distribution).


mining are introduced in [19]. In this case, the goal is toefficiently compute itemset support. Unfortunately, there isno notion of transposing results since transactions are givenin a vertical layout. Programming a clustering algorithmwith SQL queries is explored in [14], which shows ahorizontal layout of the data set enables easier and simplerSQL queries. Alternative SQL extensions to performspreadsheet-like operations were introduced in [20]. Theiroptimizations have the purpose of avoiding joins to expresscell formulas, but are not optimized to perform partialtransposition for each group of result rows. The PIVOT andCASE methods avoid joins as well.

Our SPJ method proved horizontal aggregations can beevaluated with relational algebra, exploiting outer joins,showing our work is connected to traditional queryoptimization [7]. The problem of optimizing queries withouter joins is not new. Optimizing joins by reorderingoperations and using transformation rules is studied in [6].This work does not consider optimizing a complex query thatcontains several outer joins on primary keys only, which isfundamental to prepare data sets for data mining. Tradi-tional query optimizers use a tree-based execution plan, butthere is work that advocates the use of hypergraphs toprovide a more comprehensive set of potential plans [1]. Thisapproach is related to our SPJ method. Even though theCASE construct is an SQL feature commonly used in practiceoptimizing queries that have a list of similar CASEstatements has not been studied in depth before.

Research on efficiently evaluating queries with aggrega-tions is extensive. We focus on discussing approaches thatallow transposition, pivoting, or cross-tabulation. The im-portance of producing an aggregation table with a cross-tabulation of aggregated values is recognized in [9] in thecontext of cube computations. An operator to unpivot a tableproducing several rows in a vertical layout for each input rowto compute decision trees was proposed in [8]. The unpivotoperator basically produces many rows with attribute-valuepairs for each input row and thus it is an inverse operator ofhorizontal aggregations. Several SQL primitive operators fortransforming data sets for data mining were introduced in[3]; the most similar one to ours is an operator to transpose atable, based on one chosen column. The TRANSPOSEoperator [3] is equivalent to the unpivot operator, producingseveral rows for one input row. An important difference isthat, compared to PIVOT, TRANSPOSE allows two or morecolumns to be transposed in the same query, reducing thenumber of table scans. Therefore, both UNPIVOT and

TRANSPOSE are inverse operators with respect to horizontalaggregations. A vertical layout may give more flexibilityexpressing data mining computations (e.g., decision trees)with SQL aggregations and group-by queries, but it isgenerally less efficient than a horizontal layout. Later, SQLoperators to pivot and unpivot a column were introduced in[5] (now part of the SQL Server DBMS); this work took a stepbeyond by considering both complementary operations: oneto transpose rows into columns and the other one to convertcolumns into rows (i.e., the inverse operation). There areseveral important differences with our proposal, though thelist of distinct to values must be provided by the user,whereas ours does it automatically; output columns areautomatically created; the PIVOT operator can only trans-pose by one column, whereas ours can do it with severalcolumns; as we saw in experiments, the PIVOT operatorrequires removing unneeded columns (trimming) from theinput table for efficient evaluation (a well-known optimiza-tion to users), whereas ours work directly on the input table.Horizontal aggregations are related to horizontal percentageaggregations [13]. The differences between both approachesare that percentage aggregations require aggregating at twogrouping levels, require dividing numbers and need takingcare of numerical issues (e.g., dividing by zero). Horizontalaggregations are more general, have wider applicability andin fact, they can be used as a primitive extended operator tocompute percentages. Finally, our present paper is asignificant extension of the preliminary work presented in[12], where horizontal aggregations were first proposed. Themost important additional technical contributions are thefollowing. We now consider three evaluation methods,instead of one, and DBMS system programming issues likeSQL code generation and locking. Also, the older work didnot show the theoretical equivalence of methods, nor the(now popular) PIVOT operator which did not exist back then.Experiments in this newer paper use much larger tables,exploit the TPC-H database generator, and carefully studyquery optimization.

6 CONCLUSIONS

We introduced a new class of extended aggregate functions,called horizontal aggregations which help preparing datasets for data mining and OLAP cube exploration. Specifi-cally, horizontal aggregations are useful to create data setswith a horizontal layout, as commonly required by datamining algorithms and OLAP cross-tabulation. Basically, ahorizontal aggregation returns a set of numbers instead of asingle number for each group, resembling a multidimen-sional vector. We proposed an abstract, but minimal,extension to SQL standard aggregate functions to computehorizontal aggregations which just requires specifyingsubgrouping columns inside the aggregation function call.From a query optimization perspective, we proposed threequery evaluation methods. The first one (SPJ) relies onstandard relational operators. The second one (CASE) relieson the SQL CASE construct. The third (PIVOT) uses a built-in operator in a commercial DBMS that is not widelyavailable. The SPJ method is important from a theoreticalpoint of view because it is based on select, project, and join(SPJ) queries. The CASE method is our most importantcontribution. It is in general the most efficient evaluationmethod and it has wide applicability since it can be


TABLE 7Impact of Probabilistic Distribution of Right Key

Grouping Values (N ¼ 8M)


programmed combining GROUP-BY and CASE statements.We proved the three methods produce the same result. Wehave explained it is not possible to evaluate horizontalaggregations using standard SQL without either joins or“case” constructs using standard SQL operators. Ourproposed horizontal aggregations can be used as a databasemethod to automatically generate efficient SQL queries withthree sets of parameters: grouping columns, subgroupingcolumns, and aggregated column. The fact that the outputhorizontal columns are not available when the query isparsed (when the query plan is explored and chosen) makesits evaluation through standard SQL mechanisms infeasi-ble. Our experiments with large tables show our proposedhorizontal aggregations evaluated with the CASE methodhave similar performance to the built-in PIVOT operator.We believe this is remarkable since our proposal is based ongenerating SQL code and not on internally modifying thequery optimizer. Both CASE and PIVOT evaluationmethods are significantly faster than the SPJ method.Precomputing a cube on selected dimensions produced anacceleration on all methods.

There are several research issues. Efficiently evaluatinghorizontal aggregations using left outer joins presentsopportunities for query optimization. Secondary indexeson common grouping columns, besides indexes on primarykeys, can accelerate computation. We have shown ourproposed horizontal aggregations do not introduce conflictswith vertical aggregations, but we need to develop a moreformal model of evaluation. In particular, we want to studythe possibility of extending SQL OLAP aggregations withhorizontal layout capabilities. Horizontal aggregations pro-duce tables with fewer rows, but with more columns. Thusquery optimization techniques used for standard (vertical)aggregations are inappropriate for horizontal aggregations.We plan to develop more complete I/O cost models for cost-based query optimization. We want to study optimization ofhorizontal aggregations processed in parallel in a shared-nothing DBMS architecture. Cube properties can be general-ized to multivalued aggregation results produced by ahorizontal aggregation. We need to understand if horizontalaggregations can be applied to holistic functions (e.g.,rank()). Optimizing a workload of horizontal aggregationqueries is another challenging problem.

ACKNOWLEDGMENTS

This work was partially supported by US National ScienceFoundation grants CCF 0937562 and IIS 0914861.

REFERENCES

[1] G. Bhargava, P. Goel, and B.R. Iyer, “Hypergraph BasedReorderings of Outer Join Queries with Complex Predicates,”Proc. ACM SIGMOD Int’l Conf. Management of Data (SIGMOD ’95),pp. 304-315, 1995.

[2] J.A. Blakeley, V. Rao, I. Kunen, A. Prout, M. Henaire, and C.Kleinerman, “.NET Database Programmability and Extensibilityin Microsoft SQL Server,” Proc. ACM SIGMOD Int’l Conf.Management of Data (SIGMOD ’08), pp. 1087-1098, 2008.

[3] J. Clear, D. Dunn, B. Harvey, M.L. Heytens, and P. Lohman, “Non-Stop SQL/MX Primitives for Knowledge Discovery,” Proc. ACMSIGKDD Fifth Int’l Conf. Knowledge Discovery and Data Mining(KDD ’99), pp. 425-429, 1999.

[4] E.F. Codd, “Extending the Database Relational Model to CaptureMore Meaning,” ACM Trans. Database Systems, vol. 4, no. 4,pp. 397-434, 1979.

[5] C. Cunningham, G. Graefe, and C.A. Galindo-Legaria, “PIVOTand UNPIVOT: Optimization and Execution Strategies in anRDBMS,” Proc. 13th Int’l Conf. Very Large Data Bases (VLDB ’04),pp. 998-1009, 2004.

[6] C. Galindo-Legaria and A. Rosenthal, “Outer Join Simplificationand Reordering for Query Optimization,” ACM Trans. DatabaseSystems, vol. 22, no. 1, pp. 43-73, 1997.

[7] H. Garcia-Molina, J.D. Ullman, and J. Widom, Database Systems:The Complete Book, first ed. Prentice Hall, 2001.

[8] G. Graefe, U. Fayyad, and S. Chaudhuri, “On the EfficientGathering of Sufficient Statistics for Classification from LargeSQL Databases,” Proc. ACM Conf. Knowledge Discovery and DataMining (KDD ’98), pp. 204-208, 1998.

[9] J. Gray, A. Bosworth, A. Layman, and H. Pirahesh, “Data Cube: ARelational Aggregation Operator Generalizing Group-by, Cross-Tab and Sub-Total,” Proc. Int’l Conf. Data Eng., pp. 152-159, 1996.

[10] J. Han and M. Kamber, Data Mining: Concepts and Techniques, firsted. Morgan Kaufmann, 2001.

[11] G. Luo, J.F. Naughton, C.J. Ellmann, and M. Watzke, “LockingProtocols for Materialized Aggregate Join Views,” IEEE Trans.Knowledge and Data Eng., vol. 17, no. 6, pp. 796-807, June 2005.

[12] C. Ordonez, “Horizontal Aggregations for Building Tabular DataSets,” Proc. Ninth ACM SIGMOD Workshop Data Mining andKnowledge Discovery (DMKD ’04), pp. 35-42, 2004.

[13] C. Ordonez, “Vertical and Horizontal Percentage Aggregations,”Proc. ACM SIGMOD Int’l Conf. Management of Data (SIGMOD ’04),pp. 866-871, 2004.

[14] C. Ordonez, “Integrating K-Means Clustering with a RelationalDBMS Using SQL,” IEEE Trans. Knowledge and Data Eng., vol. 18,no. 2, pp. 188-201, Feb. 2006.

[15] C. Ordonez, “Statistical Model Computation with UDFs,” IEEETrans. Knowledge and Data Eng., vol. 22, no. 12, pp. 1752-1765, Dec.2010.

[16] C. Ordonez, “Data Set Preprocessing and Transformation in aDatabase System,” Intelligent Data Analysis, vol. 15, no. 4, pp. 613-631, 2011.

[17] C. Ordonez and S. Pitchaimalai, “Bayesian Classifiers Pro-grammed in SQL,” IEEE Trans. Knowledge and Data Eng., vol. 22,no. 1, pp. 139-144, Jan. 2010.

[18] S. Sarawagi, S. Thomas, and R. Agrawal, “Integrating AssociationRule Mining with Relational Database Systems: Alternatives andImplications,” Proc. ACM SIGMOD Int’l Conf. Management of Data(SIGMOD ’98), pp. 343-354, 1998.

[19] H. Wang, C. Zaniolo, and C.R. Luo, “ATLAS: A Small ButComplete SQL Extension for Data Mining and Data Streams,”Proc. 29th Int’l Conf. Very Large Data Bases (VLDB ’03), pp. 1113-1116, 2003.

[20] A. Witkowski, S. Bellamkonda, T. Bozkaya, G. Dorman, N.Folkert, A. Gupta, L. Sheng, and S. Subramanian, “Spreadsheetsin RDBMS for OLAP,” Proc. ACM SIGMOD Int’l Conf. Managementof Data (SIGMOD ’03), pp. 52-63, 2003.

Carlos Ordonez received the degree in appliedmathematics and the MS degree in computerscience, from UNAM University, Mexico, in 1992and 1996, respectively. He received the PhDdegree in computer science from the GeorgiaInstitute of Technology, in 2000. He workedsix years extending the Teradata DBMS withdata mining algorithms. He is currently anassociate professor at the University of Houston.His research is centered on the integration of

statistical and data mining techniques into database systems and theirapplication to scientific problems.

Zhibo Chen received the BS degree in electricalengineering and computer science in 2005 fromthe University of California, Berkeley, and theMS and PhD degrees in computer science fromthe University of Houston in 2008 and 2011,respectively. His research focuses on queryoptimization for OLAP cube processing.



Horizontal aggregation for SQL

Documents

Transcript of Horizontal aggregation for SQL