Post on 17-Jul-2020
2012-2019 - Rick Flagler IT Consulting, LLC
Supporting Reporting projects with Db2
1
Rick Flagler IT Consulting, LLC Richmond, NH (M) 603-831-6873 (H) 603-239-6330 Rick.Flagler@ne.rr.com
This session will review DB2 capabilities we have to massage data, create reports and support Business Intelligence (BI). We’ll look at various Db2/SQL techniques, with advantages of each and with an eye to performance impacts. On the BI front, a consideration of many vendors with focus on IBM’s Db2 Web Query for i as a presentation layer. Differentiating the Express and Standard Edition offerings.
This presentation illustrates SQL features commonly used to facilitate reporting, data warehousing and Business Intelligence. Attendees will see numerous examples of features that can reduce HLL programming or could be used in conjunction with any BI tool of choice. Some examples demonstrated will utilize IBM Db2 Web Query for i - Express Edition (the less expensive version) for presentation of query results.
Db2 Web Query for i is a product developed by IBM in collaboration with Information Builders Inc. and targets the IBM i platform for BI, DW and ETL.
2012-2019 - Rick Flagler IT Consulting, LLC
• Understand WHAT you are reporting (key metrics)
• Determine HOW to gather data
• Decide HOW to summarize
• PLAN to achieve high-performance
• RECOMMENDATION - Obtain Executive Sponsorship when launching a BI or DW initiative.
Important aspects of Business Intelligence & Reporting
Db2 contains multiple techniques to facilitate your efforts
Regardless of tooling, better
answer these
!2
2012-2019 - Rick Flagler IT Consulting, LLC
• Views - rearranging data to present
• CTEs - Common Table expressions - make life easier When creating views. Reduce complexity. Also can be used recursively, when processing tables containing Bill-of-materials or other hierarchies.
• Use of SUBSTR, CHAR, DEC, Calculations to create or modify database fields
• Indexes - keys to performance
• OLAP - on-line analytical processing
• Ranking, summarizing
•MQTs - summarizing data to present
• An interesting cross between an index and a view
• Procedures - Encapsulating logic
• utilize parameters or apply important logic to your data
• returning result(s)
Tools in Db2 to facilitate reporting
!3
2012-2019 - Rick Flagler IT Consulting, LLC
Views - What they enable•Rearrange or select data
•columns - pick only columns that matter
•where/having clauses - limit which records you return
•Convert 6-digit date numbers to DATE data types
•Extract YEAR, MONTH,DAY from dates
•Create/Extend values that are NOT present in underlying tables
•SALES = QTY * PRICE
•DAYSTILDUE = DAYS(DueDate) - DAYS(InvoiceDate)
•Summarize data into logic groupings
•Group by SUM, AVG, MIN, MAX, more…
•Join files to denormalize for presentation and reporting
•Add descriptions for CODE fields
Reduce number of
records / fields to improve
performance
!4
2012-2019 - Rick Flagler IT Consulting, LLC
Basic View Create Syntax Create View {library}.MYTEST as ( SELECT Country, Region, state, storename, productnumber, linetotal AS Revenue FROM qwqcent . orders oJOIN qwqcent . stores s ON s.storecode = o.storecode WHERE country = 'United States' and region in ('New England','North' )ORDER BY Country, Region, state, storename) rcdfmt MYTESTR;
5
Test your SELECT then wrap with Create View. Views have NO
overhead. View can’t contain an ORDER BY clause, but you can add
when using.
select * from MYTEST order by Country, region, state, store name
Possible View uses:
select Region, sum(Revenue) from MYTEST group by Region order by Region DESC
RUNQRY MYTEST
Use in RPG program, where RCDFMT clause is important
2012-2019 - Rick Flagler IT Consulting, LLC
Simple CTE example Break logic apart using CTEs, to make complex queries easier to understand
WITH base_data AS ( SELECT Country, Region, state, storename, productnumber, linetotal AS Revenue FROM qwqcent . orders oJOIN qwqcent . stores s ON s.storecode = o.storecode WHERE country = 'United States' and region in ('New England','North' )ORDER BY Country, Region, state, storename) select * from base_data;
Several temporary results can be created with format:
With T1 as ( statement1 ), T2 as ( statement2 ), Tn as ( statements ) select * from T1, T2, …Tn …..;
32 statements maximum
?
6
Intermediate CTE results (Temp Tables) may be used in combination with others or with actual database
Tables or Views. More complex examples to follow….
2012-2019 - Rick Flagler IT Consulting, LLC
Db2 Index Advice Easy way to improve performance
1
2 3
Db2 says you might benefit from
an index and number of times advised so far
7
1. locate table 2. Select index advisor 3. Show SQL or Create
2012-2019 - Rick Flagler IT Consulting, LLC
Index Advice details for a specific Table Sorted by Times Advised Descending
Sort by any column with a click. Number of times advised in
DESCENDING sequence may help identify repeat offenders!
!8
2012-2019 - Rick Flagler IT Consulting, LLC
Want the Index Creation syntax specifics? use Show SQL
Right-Click an advised Table line to create or show the SQL needed to
create the index.
!9
2012-2019 - Rick Flagler IT Consulting, LLC
Index recommendations•Don’t simply create any index that is advised
•Understand the size of files and number of times advised before taking action
•Watch advised indexes and look for repeat offenders, sort by various columns
•When creating indexes recommend you save the SQL code in an SQL source member for later rebuilds, if required.
•Beware of too many indexes - similar to LFs, more isn’t always better, since system has to maintain any index you create..
•Yes, you can use a CREATE INDEX before the beginning of a batch run, and DROP INDEX after completion, to get benefits of the index without it being permanent. Beware of file size when doing this.
•Db2 will create some temporary indexes for you, over time, but when you IPL, these may go away. If you like it, create it permanently.
More isn’t always better
!10
2012-2019 - Rick Flagler IT Consulting, LLC
OLAPOn-line Analytical
Processing
Online analytical processing (OLAP) specifications provide the ability to return ranking, row numbering, and other aggregate function information as a scalar value in a query result.
OK, but what does that really mean?
!11
2012-2019 - Rick Flagler IT Consulting, LLC
Commonly used OLAP functions for IBM i
• RANK( )
• DENSE_RANK ( )
• ROW_NUMBER ( )
!12
2012-2019 - Rick Flagler IT Consulting, LLC
SQL Syntax for OLAP functions for IBM iwith base_data as ( select Region, state, storename, sum(linetotal) as Revenue from qwqcent . orders o join qwqcent . stores s on s.storecode=o.storecode group by Region, state, storenameunion all select Region, state, storename, sum(linetotal) as Revenue from qwqcent . orders o join qwqcent . stores s on s.storecode=o.storecode where Region = 'New England' group by Region, state, storenameorder by Region, state, storename)SELECT REGION, Storename, Revenue, RANK() OVER(ORDER BY Revenue DESC) as RANK, DENSE_RANK() OVER(ORDER BY Revenue DESC)as D_RANK, ROW_NUMBER() OVER(ORDER BY Revenue DESC) as ROW_NUM FROM base_data FETCH FIRST 10 ROWS ONLY; Useful for Top-N
reporting
!13
2012-2019 - Rick Flagler IT Consulting, LLC
Rank, Dense_Rank, Row_Number
Two lines the same; Differing function
behaviorRow_Number - Numbered 1
to 10
Dense_Rank - no skipsRank -
duplicate causes two 4’s,
skipped 5
!14
2012-2019 - Rick Flagler IT Consulting, LLC
SQL Syntax for DENSE_RANK with Partition -- Ranking within a group (region) with base_data as ( select Region, state, storename, sum(linetotal) as Revenue from qwqcent . orders o join qwqcent . stores s on s.storecode=o.storecode group by Region, state, storename order by Region, state, storename)SELECT region, state, storename, revenue, DENSE_RANK() OVER(PARTITION BY REGION ORDER BY REVENUE DESC) AS REV_RANK_IN_REG— DENSE_RANK() OVER(PARTITION BY State ORDER BY REVENUE DESC) AS REV_RANK_IN_St FROM base_datawhere Region = 'New England' ;
Partition for Ranking within a group
Might switch between one and the other? In a procedure, a
parameter could control whether ST or REG is used? Or, return both and
populate two metrics…
!15
2012-2019 - Rick Flagler IT Consulting, LLC
Ranking within a Group Using Partition
Region all the same, ranked 1 to
6.
States differ, so ranked within each.
}}
}
States differ, so ranked within each.
!16
2012-2019 - Rick Flagler IT Consulting, LLC
Adding Subtotals and Totals Grouping and With Rollup
-- Adding subtotals and totals using Grouping and with Rollup WITH base_data AS (SELECT Country, Region, state, storename, linetotal AS Revenue FROM qwqcent . orders o JOIN qwqcent . stores s ON s.storecode = o.storecode where country in ('Canada' , 'United States') ORDER BY Country, Region, state, storename) SELECT ROW_NUMBER() OVER(ORDER BY Country, Region, state, storename) AS ROW_ID, Country, Region, state, storename, Sum(Revenue) as Rev, GROUPING(Country) AS GrandLvl from base_data GROUP BY Country, Region, state, storename WITH ROLLUP;
WITH ROLLUP for Reporting Subtotals
Column GrandLvl will be 0 or 1 depending upon
whether on a grand total line
WITH ROLLUP adds subtotal and total lines
!17
2012-2019 - Rick Flagler IT Consulting, LLC
GROUPING and WITH ROLLUP
Subtotals inline
Subtotal or Total indicator field ….
……Some lines omitted
!18
2012-2019 - Rick Flagler IT Consulting, LLC
Materialized Query Table (MQT)• Useful for:
• Data Warehouses (DW)
• Business Intelligence (BI)
• Staging data for some transfer
• Facilitates performance improvements (improved response) for complex queries. Particularly those featuring:
• Aggregated (summarized) data
• Joined data with several tables
• Commonly accessed data
!19
2012-2019 - Rick Flagler IT Consulting, LLC
Materialized Query Table (MQT) • Rick’s definition - IBM method to define a table often based on other table(s),
with indexes or views over it, to potentially summarize, refreshed periodically.
• Excerpt of IBM’s definition - A materialized query table (MQT) is a table whose definition is based upon the result of a query. The data that is contained in an MQT is derived from one or more tables on which the materialized query table definition is based. You can think of an MQT as a kind of materialized view. Both views and MQTs are defined on the basis of a query. The query on which a view is based is run whenever the view is referenced; however, an MQT actually stores the query results as data, and you can work with the data that is in the MQT instead of the data that is in the underlying tables. Materialized query tables can significantly improve the performance of queries, especially complex queries. If the optimizer determines that a query or part of a query could be resolved using an MQT, the query might be rewritten to take advantage of the MQT. Source IBM Developer Article 2005
!20
2012-2019 - Rick Flagler IT Consulting, LLC
Though MQTs can be directly specified in a user’s query, their real power comes from the query optimizer’s ability to recognize the existence of an appropriate MQT implicitly, and to rewrite the user’s query to use that MQT. The query accesses the MQT (instead of accessing one or more of the specified tables). This shortcut can drastically minimize the amount of data read and processed (see Figure 1).
!21
Source IBM - MQT documentation
2012-2019 - Rick Flagler IT Consulting, LLC
Cool things about MQTs• Created once, used repeatedly
• Maintained by System or by user
• Can be refreshed (or rebuilt) by the SQL command
• REFRESH TABLE{table name}
• Indexes over MQTs are not removed when refreshed
• Views over MQTs are not destroyed when table is refreshed
• Performance of MQTs vs. other Query techniques can produce significant improvement (i.e. reduce response time)
!22
2012-2019 - Rick Flagler IT Consulting, LLC
A full source MQT Example
!23
CREATE TABLE {library} . ORDMQT AS ( WITH base_data AS ( SELECT ordernumber, orderdate, requestedshipdate, shipdate, invoicedate, o.storecode, storename, plantcode, salesrep, quantity, linetotal, costofgoodssold, returns, warrantyexp, shippingcost, country, region, state, city, o.productnumber, producttype, productcategory, productname, model FROM qwqcent . orders o JOIN qwqcent . stores s ON s.storecode = o.storecode JOIN qwqcent . inventory p on o.productnumber = p.productnumber where country in ('Canada' , 'United States') --and region in (/*'Mountain',*/'New England', 'North') ORDER BY Country, Region, state, storename ) SELECT bd.*, quantity*linetotal as ExtRev,quantity*costofgoodssold as Extcost, (quantity*linetotal) - (quantity*costofgoodssold) as Extprofit, days(shipdate) - days(requestedshipdate) as OTD, case when days(shipdate) - days(requestedshipdate) < -9 then 'Very Early' when days(shipdate) - days(requestedshipdate) between -9 and -2 then '1Wk Early' when days (shipdate) - days(requestedshipdate) between -1 and +1 then 'On-time' when days (shipdate) - days(requestedshipdate) between +2 and +9 then '1Wk Late' else 'Very Late' end as OTD_Sts, case when days(shipdate) - days(requestedshipdate) < -9 then '1' when days(shipdate) - days(requestedshipdate) between -9 and -2 then '2' when days (shipdate) - days(requestedshipdate) between -1 and +1 then '3' when days (shipdate) - days(requestedshipdate) between +2 and +9 then '4' else '5' end as OTD_PER from base_data bd ORDER BY Country, Region, state, storename ) DATA INITIALLY DEFERRED
REFRESH DEFERRED MAINTAINED BY USER ENABLE QUERY OPTIMIZATION RCDFMT OrdMQTR ;
REFRESH used after 1st CREATE to load data
2012-2019 - Rick Flagler IT Consulting, LLC
Refreshing the MQT to pick up data changes to underlying
tables
!24
REFRESH table {library}. ordmqt;
REFRESH causes the MQT query to be run to refresh data in the table
Might be employed where IMMEDIATELY current data is
not required so, for example, a DAILY refresh is OK.
2012-2019 - Rick Flagler IT Consulting, LLC
Example-1 Query using MQT
!25
with a as ( select year(shipdate) as TOTYR, count(*) as totcnt from Ordmqt group by year(shipdate) ) select year(shipdate) as YR, OTD_PER, otd_sts, count(*) NbrShips, totcnt, -- decimal percent rounded to 2-places makes 100% dec(round(((count(*)*100.00)/totcnt),2),5,2) as OTD_PCT from Ordmqt join a on year(shipdate)=TOTyr group by TOTCNT, otd_sts, otd_per, year(shipdate) ORDER BY YEAR(SHIPDATE) DESC, OTD_PER;
2012-2019 - Rick Flagler IT Consulting, LLC
Example-2 Query using MQT
!26
with a as ( select year(shipdate) as YR, productname, count(*) as totcnt from Ordmqt group by productname,year(shipdate) ) select year(om.shipdate) as YR, om.productname, OTD_PER, otd_sts, count(*)Nbrships, totcnt, -- decimal percent rounded to 2-places makes 100% dec(round(((count(*)*100.00)/totcnt),2),5,2) as OTD_PCT from Ordmqt om join a on om.productname=a.productname and year(om.shipdate)=a.YR group by TOTCNT, otd_sts, otd_per, om.productname,year(om.shipdate) ORDER BY YR,productname, OTD_PER;
2012-2019 - Rick Flagler IT Consulting, LLC
MQT creation actions• Create the MQT
• Initial Refresh, then schedule Refresh actions as desired
• run queries against the MQT data
• Add indexes or Views to support various Queries you run or anticipate
• Examine Index Advisor for table periodically
!27
2012-2019 - Rick Flagler IT Consulting, LLC
Business Intelligence
!28
Every vendor will say: • Their product is best • Try it for free • Get your BI/DW project started in 90-minutes • We can do it in the Cloud, etc. etc.
Not sure ALL claims are true but reporting clearly isn’t as hard as it used to be!!!
Probably two big players with “Native” products on IBM i: IBM - Db2 Web Query for i - $ or $$ Information Builders - Webfocus - $$$ or $$$$
Many vendors, Many price points
2012-2019 - Rick Flagler IT Consulting, LLC
Business Intelligence
!29
Chart at right -> Sourced from:
2012-2019 - Rick Flagler IT Consulting, LLC
Why use BI tools on top of Db2 for i?• Db2 is best in class database • Your data is already there in Db2, why do BI elsewhere? • BI tools offer more sophistication than QUERY/400 product• Present data visually - “A picture is worth 1000 words!”• Modern tools are designed for Self-service by users rather than by
programmers• Typically “data setup” is required by a DBA, analyst or QPGMRs• The tools CODE many features for you, often automating the
SQL and creating intermediate or final totals• Enables easy Pivot and Drill-Down of data spanning Time,
Geography or Family• OLAP features enable DRILL-DOWN from one level to another
• Free up programmers for high-value tasks rather than saddle with report programming
!30
2012-2019 - Rick Flagler IT Consulting, LLC2012-2019 - Rick Flagler IT Consulting, LLC
TYPICAL BI ENVIRONMENT
31
IBM POWER System Hardware
Db2 DatabaseWeb/Http Server
AR SALES IT PURCH FIN MFG
Enterprise Data
Reports written using BI
Other BI tool
OPER
IBM Db2 Web Query for i
Business Functional Areas
For IBM i
2012-2019 - Rick Flagler IT Consulting, LLC
Features most BI tools offer•Widgets in the form of Pie, Bar, Surface, Area, Line Charts,
Geographic Mapping and other report types
•Development using WSYIWYG designer tools, Browser or Client based
•MetaData which enables you to enhance what Db2 tables have without touching the Db2 tables. Examples:
• Rename fields to more meaningful names
• Convert legacy formatted dates to DATE data types
• Join files so users don’t have to worry about bad joins
•Ability to pass parameters between reports
•Drill-down linkages between related subject matter
•A repository of stored reports with analysis about usage
•Security features to implement allowed viewing !32
2012-2019 - Rick Flagler IT Consulting, LLC
Report typesLine - Trend Pie - current breakdown Metric - Indicator
Bar - ComparisonReport - Expandable
Report - Traffic Lighting
Map - Geographic penetration
2012-2019 - Rick Flagler IT Consulting, LLC
TYPICAL CHARTS/GRAPHS
34
Bars - Compare one group to anotherPie - Compare size/magnitudeLine or Area - Plot values/progress over time
You need to understand your data and sometimes limit the data or reformat in SQL or in BI tools with calculations.
Did you know? If you study Lean-6-Sigma, statistics experts DISLIKE pie charts, saying they do not convey enough information.
2012-2019 - Rick Flagler IT Consulting, LLC
EXAMPLE OF PICKING WRONG
35
Chart axis is plotting every date in a many record file - too much data spoils the chart.Solutions? - Plot by month, quarter, year or prompt for a limited range to plot
2012-2019 - Rick Flagler IT Consulting, LLC 36
PIES AND GAUGES
In context of a Visualization report for analysis
2012-2019 - Rick Flagler IT Consulting, LLC2012-2019 - Rick Flagler IT Consulting, LLC
CAVEATS ABOUT BROWSER-BASED REPORTING
• Most BI tools allow you to quickly create reports in Web Browser formats
• But BI is not about putting today’s 300 page report in a Web Browser (or on paper) from your BI tool
• Business Intelligence is more about:
• Metrics• Drill-down• Data Exploration• Analysis• Trend spotting
37
Recommendations:• Define Key Performance Indicators• Capture metrics• Chart trends• Create dashboards, OLAP, Drill-down
2012-2019 - Rick Flagler IT Consulting, LLC
IBM Db2 Web Query for i
!38
Express Edition Standard Edition
Express Edition
Report Scheduleing and Delivery
Commands
More DB Adapters
Three offerings currently
Repository Security
Named Users
Data Migrsator - ETL tool
Report Design Report Execution
OLAP Drill-down linkages
Charts, Maps, Visualization
Run-time Users
You can give ALL of this a try on your IBM i system with a FREE 70-day trial
of IBM Db2 Web Query for i
1 2
3
2012-2019 - Rick Flagler IT Consulting, LLC
Stored Procedures• Programs written in SQL language (or in your favorite HLL)
• Doesn’t run from IBM i Command line, but is executed like other SQL from:
• STRSQL
• Client Access for Windows - Run SQL Scripts
• RUNSQL command
• RUNSQLSTM command
• ACS - Run SQL Scripts
• QSHELL - QSH Db2utility
• From PHP or other open source Languages with DB2, ODBC or JDBC connectivity
!39
2012-2019 - Rick Flagler IT Consulting, LLC2012-2019 - Rick Flagler IT Consulting, LLC
USING STORED PROCEDURES
40
• Advantages• Powerful• Drives business rules into database (Db2)• Provide customized parameter-driven queries• Return one or more Results• Performance benefits• One line of SQL = 50-100 lines of RPG or other HLL• Hides complexity
• Disadvantages (if any?)• Requires additional SQL knowledge and programming skills
Programming in the SQL language
Also, can “wrap” existing RPG logic in SQL to make RPG logic reusable in your reporting Applications
2012-2019 - Rick Flagler IT Consulting, LLC
Where to build/test your SQL• Green screen STRSQL - POSSIBLE BUT HARD
• Client Access for Windows - NOT any more - WITHDRAWN by IBM this year for Win10 and higher
• Access Client Solutions (ACS) - YES = The new Client
• Runs on Mac, Linux, Windows 8 and above
• IBM Data Studio - MAYBE = If you have it
• Rational Developer for i - YES = at least to code and view
!41
2012-2019 - Rick Flagler IT Consulting, LLC
Steps to construct a SP to drive reporting•Identify file(s) used
•Create individual SQL statements to verify data extraction and summarization from ERP system data
•Create SP using your favorite IBM i SQL tool
•Declare SQL statements with individual result sets
•Test execute SP in script to verify it generates one, two, …n result sets
•In Web Query or other BI tool, configure your report to consume the result data
•Repeat - Creating a number of charts, reports, graphs from various results
•Save and Run!42
2012-2019 - Rick Flagler IT Consulting, LLC
A few Caveats• The following examples:
• Were developed using ACS - Run SQL Scripts
• SQL techniques should work on IBM I 7.1 and above
• SQL techniques should work for any presentation technique with any BI tool or open-source charting method
• Assumption here is results will be consumed with IBM Db2 Web Query for i
• Using the IBM-Supplied WQ sample data in library QWQCENT
• Provides YOU with a framework for similar reporting
!43
2012-2019 - Rick Flagler IT Consulting, LLC
SQL example selectWITH dt as (
select LAST_DAY(max(orderdate))-12 months as begin_date FROM qwqcent.orders )
SELECT STORECODE, SUM(LINETOTAL * QUANTITY) AS REVENUE FROM qwqcent.orders , dt where orderdate > begin_date GROUP BY STORECODE ORDER BY REVENUE DESC FETCH FIRST 10 ROWS ONLY;
With denotes use of a Common Table Expression (CTE)DT is a temporary result table for this query, containing a BEGIN_DATE computed to be 12 months prior to last day of the month of the most recent order date in the subject file..
The second select joins the temp table DT to the original ORDERS table to provide 1-year of data while also summarizing to yield the top-10 stores by revenue descending. This result can be returned rapidly to facilitate a chart or table of information.
** Depending on your data, the date calculation in DT might require CURRENT DATE to be used, from which you could back off a set period or days, months, years with SQL date math.
!44
2012-2019 - Rick Flagler IT Consulting, LLC
SQL example selectWITH a AS ( SELECT YEAR(orderdate) AS ordYY, MONTH(orderdate) AS ordM, (LINETOTAL * QUANTITY) AS Period_REVENUE, (COSTOFGOODSSOLD * QUANTITY) AS Period_COST FROM qwqcent.orders ) SELECT ordYY || '-' || SUBSTR(DIGITS(ordM), 9, 2) as Period , sum(Period_REVENUE - Period_COST) AS PerProfit FROM a GROUP BY ordYY || '-' || SUBSTR(DIGITS(ordM), 9, 2) ORDER BY ordYY || '-' || SUBSTR(DIGITS(ordM), 9, 2) DESC FETCH FIRST 12 ROWS ONLY
With denotes use of a Common Table Expression (CTE)A is a temporary result table for this query, extracting the year and month from a date, computing revenue and cost of goods.
The second select uses the temp table A to summarize by Period, descending fetching the most recent 12 months which can be charted as a line or bar chart, to provide 1-year of historical data. Yes, I could have constructed column Period in the first select.
!45
2012-2019 - Rick Flagler IT Consulting, LLC
SQL to DECLARE a named cursor for select in an SP
DECLARE C1 CURSOR FOR WITH dt as ( select LAST_DAY(max(orderdate))-12 months as begin_date FROM qwqcent.orders ) SELECT STORECODE, SUM(LINETOTAL * QUANTITY) AS REVENUE FROM qwqcent.orders , dt where orderdate > begin_date GROUP BY STORECODE ORDER BY REVENUE DESC FETCH FIRST 10 ROWS ONLY;
DECLARE is only allowed in SQL program (function or Procedure).
Cursor C1 is defined by the DECLARE and can later be used to fetch records or to pass data back to the calling application.
!46
2012-2019 - Rick Flagler IT Consulting, LLC
SQL to Create a procedure -- Define procedureCREATE PROCEDURE Library.ProcName () DYNAMIC RESULT SETS 1 — or more than 1 LANGUAGE SQL BEGIN—Declare the Statement(s) DECLARE C1 CURSOR FOR WITH dt as (
select LAST_DAY(max(orderdate))-12 months as begin_date FROM qwqcent.orders )
SELECT STORECODE, SUM(LINETOTAL * QUANTITY) AS REVENUE FROM qwqcent.orders , dt where orderdate > begin_date GROUP BY STORECODE ORDER BY REVENUE DESC FETCH FIRST 10 ROWS ONLY;--Open each Result set(s) OPEN C1;— Set results and return (END) to caller SET RESULT SETS CURSOR C1; END; One or more DECLARE statements enables you to return several results to the calling application. If parameters are required, they would be inserted after ProcName ( specifying parameter & datatype i.e. P1 CHAR(5) , P2 DEC(9,0) ) and subsequently those variables could be utilized within the procedure's logic.
!47
2012-2019 - Rick Flagler IT Consulting, LLC
Typical development Approach for Stored Procedure
• Test SELECT statement independently to verify it returns the proper information for your application
• Wrap the select with DECLARE {name} CURSOR statement
• Wrap DECLARE with CREATE PROCEDURE statement
• Add OPEN, SET RESULTS and END
• RUN Create Procedure to use SQL source to generate a program
• Run Procedure via CALL [Procedure name}
• BOOM! - you have data results
!48
2012-2019 - Rick Flagler IT Consulting, LLC
Full SQL for Example procedure with 5 results to drive BI example-- Define procedure with 5 results CREATE OR REPLACE PROCEDURE Dash2() DYNAMIC RESULT SETS 5 LANGUAGE SQL BEGIN
DECLARE C1 CURSOR FOR WITH dt as ( select LAST_DAY(max(orderdate))-12 months as begin_date FROM qwqcent.orders ) SELECT STORECODE, SUM(LINETOTAL * QUANTITY) AS REVENUE FROM qwqcent.orders , dt where orderdate > begin_date GROUP BY STORECODE ORDER BY REVENUE DESC FETCH FIRST 10 ROWS ONLY; DECLARE C2 CURSOR FOR WITH dt as ( select LAST_DAY(max(orderdate))-12 months as begin_date FROM qwqcent.orders ) SELECT PLANTCODE, SUM(LINETOTAL * QUANTITY) AS REVENUE FROM qwqcent.orders, dt GROUP BY PLANTCODE ORDER BY REVENUE DESC FETCH FIRST 10 ROWS ONLY; DECLARE C3 CURSOR FOR WITH dt as ( select LAST_DAY(max(orderdate))-12 months as begin_date FROM qwqcent.orders ) SELECT SALESREP, SUM(LINETOTAL * QUANTITY) AS REVENUE FROM qwqcent.orders ,dt GROUP BY SALESREP ORDER BY REVENUE DESC FETCH FIRST 10 ROWS ONLY; DECLARE C4 CURSOR FOR WITH a AS (SELECT YEAR(orderdate) AS ordYY, MONTH(orderdate) AS ordM, (LINETOTAL * QUANTITY) AS Period_REVENUE, (COSTOFGOODSSOLD * QUANTITY) AS Period_COST FROM qwqcent.orders) SELECT ordYY || '-' || SUBSTR(DIGITS(ordM), 9, 2) as Period , sum(Period_REVENUE - Period_COST) AS PerProfit FROM a GROUP BY ordYY || '-' || SUBSTR(DIGITS(ordM), 9, 2) ORDER BY ordYY || '-' || SUBSTR(DIGITS(ordM), 9, 2) DESC
FETCH FIRST 12 ROWS ONLY ;
DECLARE C5 CURSOR FOR WITH dt as ( select LAST_DAY(max(orderdate))-12 months as begin_date FROM qwqcent.orders ), a AS ( SELECT YEAR(orderdate) AS ordYY, MONTH(orderdate) AS ordM, case when days( shipdate ) - days(requestedshipdate) between -2 and +2 then 'On-time' when days( shipdate ) - days(requestedshipdate) < -2 then 'Early' when days( shipdate ) - days(requestedshipdate) > +2 then 'Late' end as status , 1 as total FROM qwqcent.orders,dt where orderdate > begin_date ) SELECT ordYY || '-' || SUBSTR(DIGITS(ordM), 9, 2) as Period , sum(case status when 'Early' then 1 else 0 end ) as Early, sum(case status when 'On-time' then 1 else 0 end ) as ontime, sum(case status when 'Late' then 1 else 0 end ) as Late, sum(total) as total, dec((sum(case status when 'On-time' then 1 else 0 end ) *100.0) / sum(total),9,2) as OTDPct FROM a GROUP BY ordYY || '-' || SUBSTR(DIGITS(ordM), 9, 2) ORDER BY ordYY || '-' || SUBSTR(DIGITS(ordM), 9, 2) desc; --Open each of the Result sets and return them OPEN C1; OPEN C2; OPEN C3; OPEN C4; OPEN C5; SET RESULT SETS CURSOR C1, CURSOR C2, CURSOR C3, CURSOR C4, CURSOR C5; END;
!49
2012-2019 - Rick Flagler IT Consulting, LLC
SAMPLE “DASHBOARD” DRIVEN BY STORED PROCEDURE
50
Run many metrics simultaneously, Improve dashboard performance
SP gives several results, each returns recent data, plotting reports or charts.
2012-2019 - Rick Flagler IT Consulting, LLC
Reasons you might take the SP approach
• All logic is located in one place
• Runs fast because of FETCH limiting number of records
• DB2 optimizer probably “remembers” after SP runs several times, creates “plan”
• Easier to document and explain to others
• Easy to clone for future applications
!51
2012-2019 - Rick Flagler IT Consulting, LLC
How the SP data is used in Web Query1.Select data source - DASH2 in
this case, Using procedure name yields available fields
2.See various fields from each ANSWERSET which correspond to the SP result sets
3.Drag fields to report definition(s)
4.Press RUN to see data summarized in report or chart
1
2
3
4
!52
2012-2019 - Rick Flagler IT Consulting, LLC
Web Query InfoAssist Designer window with multiple reports on single canvas
As you design each “report” you can see different fields used. When Run, all SP values
are returned to populate the canvas areas.
Available Fields Report
DefinitionCanvas
workspace!53
2012-2019 - Rick Flagler IT Consulting, LLC
Capturing Enterprise Data for BI reporting
• Copy data from IBM i to Excel
• Copy data off IBM i to a Windows/Linux Server
• Copy data to another LPAR where BI tool resides
• Utilize Db2 objects and programming to reduce burden of reporting on Transactional machines. Make summary data available quickly to enable responsive applications, with drill down and analysis capabilities.
My rankings of approaches …. your opinion may vary!!
!54
2012-2019 - Rick Flagler IT Consulting, LLC
New Web Query Visualizations offer another option with or without Stored Procedures
!55
This example uses tables joined by the WQ metadata Redraws when user modifies selection filters
Filters for What-if analysis
2012-2019 - Rick Flagler IT Consulting, LLC
Auto Drill down exampleLinkage - Develop single report, get automatic drill-down to levels in a
pre-defined hierarchy
Click to drill-down, Click to return
2012-2019 - Rick Flagler IT Consulting, LLC
Other Reporting Techniques • IBM provides SERVICES to access various system information via SQL
• Jobs, Subsystems, Users
• Spool Files
• Consider creating your own “services” for use in business reporting or to support BI projects, using:
• User-Defined table functions (UDTF)
• Views
• MQTs
• Such “Services” might expose data so users can make their own reports
!57
2012-2019 - Rick Flagler IT Consulting, LLC
One last approach to Enterprise metrics
!58
2012-2019 - Rick Flagler IT Consulting, LLC
Creating an Enterprise metric database•Requires some brain-storming and discussion to determine what Management wants
•Metric “data points” are harvested and stored for reporting
•Small batch jobs on your Job Scheduler will populate metrics by doing summarize and insert for recent time period
•Possible examples:
•Sales $ vs. Forecast $
•On-Time Delivery % attainment
•Year-over-year Value comparisons
•Quality objective attainment (sales vs. scrap)
•Lost-time Accident metrics vs. Days since last accident
•Inventory turns
•Inventory Days on Hand!59
2012-2019 - Rick Flagler IT Consulting, LLC 60
Example metric database design
Example data exposed by View
Records inserted into detail by night-time jobs, reference definitions for metric and report
2012-2019 - Rick Flagler IT Consulting, LLC 61
Example of individual metrics plotted
Create reports that present last X months of
2012-2019 - Rick Flagler IT Consulting, LLC
SUMMARY ADVICE• Drive Work into Db2
whenever possible
• Present data visually when possible
• Drill down to dimension details in order to reduce results presented
• Pick the right chart type for presentations
• Don’t try to print the world to your web browser
• Use Color to differentiate values
• via Traffic-lighting• via data bars• Charts
• Don’t over-use color in Themes and Styles
• Use Stored procedures or User Defined Table functions if they fit your requirements
• Build a metric database to provide data points for frequently used KPIs
62
2012-2019 - Rick Flagler IT Consulting, LLC2012-2019 - Rick Flagler IT Consulting, LLC
END OF PRESENTATION
63
QUESTIONS AND DISCUSSION