Data Preparation in theQuadstone System Version 5

Thursday, 17th February 20057.30am PST / 10.30am EST / 3.30pm GMT / 16.30 CETPlease join the teleconference call now; if you have any

difficulty, contact support@quadstone.com.

Starting in 15 minutesStarting in 10 minutesStarting in 5 minutesStarting in 2 minutesStarting now

© 2005 Quadstone

How to ask questions

• Return to WebEx Event Manager:• Use Q&A (not Chat):

• You can return to full-screen view:

© 2005 Quadstone

Data Preparation in the Quadstone System V5

• Presenter: Joshua Lewis, Quadstone Consultant • Overview: Interactive data-preparation: sorting,

aggregating, joining, and deriving• Audience: Experienced Quadstone System users

looking to undertake ad-hoc preparation of data• Format:

• A live demo with slides for sign-posting• Follow-up exercises in the form of a workbook and dataset

• Duration: 1 hour, including Q&A

© 2005 Quadstone

Interactive data preparation

• Wizards for each action• Right-click in Quadstone System

Explorer (QSE), or• Click on (or drag file to) Quadstone

System Shortcut Bar

• Choose parameters as appropriateBest practice: keep an audit trail

© 2005 Quadstone

Transactiondata

Customerdata

To be filled

Analysis dataset

Customer IDs

A simple data preparation process

SORT SOR

TMEASUR

EJOIN

DERIVE

DERIVE

© 2005 Quadstone

Join Fields adds fields from a secondary focus to a primary focus.

Pre-requisites:• Key fields in both foci, of the exact same datatype• The records in both foci must be sorted by the key

fields

• Subtly different from Import Fields in Decisionhouse

Joining foci in QSE

© 2005 Quadstone

Field derivation

22/10/1998 09/05/199901/09/199604/11/199702/03/199522/01/1995...

NULLNULL49435272

...

112111

...

CustomerID StartDate Age Gender

163004187006188008190006268006 36004...

22/10/1998 09/05/199901/09/199604/11/199702/03/1995

NULLNULL494352

11211

CustomerID StartDate Age Gender MonthsTenure

163004187006188008190006268006

Derivation

103352153

© 2005 Quadstone

Deriving new fields

• Requires:• A focus (need not be sorted)• A derivations (.tml) file containing TML

descriptions of the fields to be derived

• Create a new TML file via right-click in QSE, then right-click again to Edit

Best practice: Develop and debug the FDL expressions interactively in Decisionhouse, before embedding them within the TML file

© 2005 Quadstone

Derivation syntax

create Tenure := countwholemonths(StartDate,today());create YoungMan := Age < 30 and Gender = 1;

• Creates one output record per input record• The first example counts the number of months since a

person became a customer; the second creates a flag to identify a specific segment of customers

• General syntax:create <fieldname> := <FDL expression> ;

• See online help: Field Derivation Language (FDL) reference

© 2005 Quadstone

Sorting foci

• Required when combining foci and grouping records

• Sort will usually, but not always, be on customer ID

Best practice: check sort order first, to see if a sort is needed

Best practice: sort once, upstream (adding keys if needed), to minimize time-consuming re-sorting

© 2005 Quadstone

Transaction measurement

05/02/199807/02/199808/02/199802/02/199805/03/199803/02/1998...

ATMATMSDSDSDSD...

30.0054.4283.8029.4940.0046.26...

CustomerID Date TransType Value

643400000064340000006434000000643400000164340000016434000002...

08/02/199805/03/199803/02/1998...

643400000064340000016434000002…

ATMSDSD...

56.0734.7546.26...

CustomerID MostRecentDate MostFreqTrans AverageValue

Rollup(aggregation)

© 2005 Quadstone

Simple aggregation

• Requires:• A focus (sorted by the grouping key

field, e.g., CustomerID)• Selection of an appropriate key field (for

grouping records)• An aggregations (.tml) file containing

TML descriptions of the aggregationsIgnore the Functions and Statistics

optionsExample TML files are in the online

help and the ext/demo/dbc folder of your installation

© 2005 Quadstone

Aggregation syntax

create NumberOfPurchases := count(); create ValueOfPurchases := sum(Amount);

• Processes each group of transaction records that share the same CustomerID (grouping key), to create one output record per CustomerID

• The first example counts the number of transactions for each customer; the second sums the values in the Amount field for each customer

• General syntax:create <fieldname> := <aggfn>( <arguments> );

• See online help: Transaction Measurement Language (TML) reference

© 2005 Quadstone

Joining foci

ID AgeA 56B 23C 31

Customers.ftrID TotalVisitsA 3C 2D 4

Visits.ftr

Customers.ftr

ID Age TotalVisitsA 56 3B 23 NullC 31 2

Join on ID

Sorted Sorted

© 2005 Quadstone

Combining datasets

Append fields: abut equal-length datasets

Join fields: match on common key(s)

Merge records: interleave using common key(s)

+ =

+ =

+ =

© 2005 Quadstone

Metadata

• You can import metadata from a previous dataset using a template focus

• Includes all derivations, selections, binnings, interpretations and comments

• Allows re-use of metadata developed interactively in Decisionhouse

• You can import metadata in XML form• Allows metadata (e.g. a data dictionary) to be

maintained externally (converted to XML)• Currently supports comments only

© 2005 Quadstone

MEASURING

qsderiveqsmeasureqsmeasuretrack

Advanced data preparation

FOC

US

qsbuild

COMBINING

qsjoinqsappendfieldsqsmerge

IMPORTING

qsdbaccessqsimportdb

qsgenfddqsimportflat

qsimportfocus

REPORTING

qsdescribeqsauditqsdtsnapshotqsscsnapshotqsxtqsxt2spec[qsinfo]

EXPORTING

qsdbcreatetableqsdbinsertqsdbupdate

qsexportflat

MANAGING

qscopyqslinkqsmoveqsremove

FOC

US

TRANSFORMING

qssort qsrenamefieldsqsselect

ENHANCING

qsimportmetadataqsupdate[qsinterp][qsexportmetadata]

qstmlqssettings

• Advanced TML aggregation syntax (filter, split)• Data Build Commands• Data Build Manager

© 2005 Quadstone

Where to find out more

• Quadstone System Help; for example:• Working with flat files, database tables, and foci• Transaction Measurement Language (TML) reference• Field Derivation Language (FDL) reference

• Quadstone System data-build command and TML reference• Examples of TML

• ext/demo/dbc folder of your installation• More example TML and data

• Quadstone System Support website: http://support.quadstone.com/

• Advanced Data Preparation training course: contact support@quadstone.com

© 2005 Quadstone

Questions and answers

© 2005 Quadstone

After the webinar

• These slides, a workbook and data are available via www.quadstone.com/training/webinars/

• Any problems or questions, please contact support@quadstone.com

© 2005 Quadstone

Upcoming webinars

See www.quadstone.com/training/webinars/.If there’s a webinar topic you’d like to see, please let us know via

support@quadstone.com.

Pragmatic Scorecarding

March 17, 2005 14:00 UK/Ireland, 15:00 Central European, 9am Eastern

Pragmatic Scorecarding

March 18, 2005 9am Pacific, 11am Central, 12noon Eastern, 5pm UK/Ireland

The Quadstone Portal

April 14, 2005 14:00 UK/Ireland, 15:00 Central European, 9am Eastern

The Quadstone Portal

April 15, 2005 9am Pacific, 11am Central, 12noon Eastern, 5pm UK/Ireland

© 2005 Quadstone

Your feedback

Please email support@quadstone.com