Post on 21-Jan-2016
description
Business Unit Empowerment Through Declarative Systems
Why IBM DB2 PureXML™ Rocks!Dan McCreary
Dan McCreary & AssociatesSeptember 11, 2007
M
D
Metadata Solutions
2M
D
DISCLAIMER
• The following views are my own and do not necessarily reflect the views of the Minnesota Department of Revenue or the State of Minnesota
3M
D
Presentation AbstractMany non-programmers feel that complex Java and .Net middle-tier systemprevents business units from being able to quickly build and modify simpleweb applications (aka CRUD applications). DB2 version 9 has powerfulsupport for quickly storing and searching complex XML documents. Thisfeature combined with storing XML in a web browser using XForms makesmiddle-tier code unnecessary, putting non-programmers in control ofapplications.
This presentation will cover a case study that allows a business unit to"draw pictures" to build simple web applications. These pictures (XMLSchema diagrams) are transformed directly into XForms and stored in DB2version 9. Rapid searching is provided using XQuery. Very little (ifany) procedural code is involved in this process. The presentationfinishes with a set of 10 recommendations that empower business units tobuild and maintain their own applications and leverage the power of acentral DB2 database.
This presentation is appropriate for business and data strategists as wellas DBAs and application architects.
4M
D
Agenda
• How did we get here?– Why trees not tables?
• Demos– NIEM, XML Schemas, XForms
• Costs of translation from XML to OO to tables
• XQuery in DB2• DB2 and your Enterprise Technology
Strategy
5M
D
Incoming!
XMLHas this webthing gone away yet?
web
6M
D
More New XML Data is Being GeneratedThan Relational Data
• Growing at 2X rate of total database market [IDC]
• XML is now pervasive in many of organizations• Almost every sector has XML based standards
"XML needs its own entry here to reflect its transformation of information … XML isenabling the revolution of Web services and driving a database revolution."-- eWeek "25 More Top Technologies", September 18, 2006
7M
D
A Happy Partnership
XFormsDB2
PureXML
8M
D
Strategic Consequences
• Empowerment of business units to build and maintain their own applications with much less involvement of IT staff
• Dramatically reduced need for “middleware” IT staff using traditional procedural languages like Java and C#
• Evolution of DB2 to a web service data providor and core component of ESB
9M
D
Did you Know…
• That with XForms you can build rich-client web applications without writing a single line of JavaScript?
• That IBM DeveloperWorks has almost 300 articles that discuss XForms?
• That DB2 version 9 "PureXML" has fantastic support for native XML data types and supports W3C XQuery standards?
10M
D
How did we get here?• From punch cards…
to tables…
to trees…
to cubes and graphs…
11M
D
In the Beginning…There were puchcards…
…and there were 80 columns…and the widths were fixed…and batches of cards were called "decks“…and they were loaded into tables in “core” memory…
12M
D
And Codd and Date said…• Let there be relationships…
Personperson_idfirst_namelast_namebirth_date
Projectproject_idproject_nmdescriptionstatus
Role
role_nm
person_idproject_id
and we had the relational databases…and we had SQL…and we had joins…and we had ER diagrams…and we had PowerBuilder…and all seemed right with the world…until
time_pct
13M
D
Sir Berners-Lee Said…• Let there be HTML…
<html> <head> <title>Dan’s Homepage</title> </head> <body> <h1>Welcome!</title> <p>Welcome to my web site!</p> <a href="about.htm">About Me</a> </body></html>and we had URIs…
and we had HTTP…and we had we links and resources…and we had trees (not tables)…and we had search…until
14M
D
Customers Said..• We wanted to access our tables…on the
web…in HTML!html
head body
h1 p a
and there were objects…and there was Java and JSP … and there was ASP.Netand there were models, views, controllers…and there was conversion from tables to objects to trees and trees to objects to tables…and there were rooms and rooms of programmers…
?
15M
D
And there Was Chaos…
html
head body
h1 p a
HTML Forms
Objects
OR Mapping
JSP/ASP
100+ Frameworks
XML
.Net
JDBC
Serialization
JavaScript
AJAX
Hibernate
And the number of languages you need to learn and the headcount of the IT departmentwent up…and up
Struts
Sessions
SQL
Stored Procedures
RDBMS
16M
D
The Translation “Pain Chain”
• From web forms to objects…to SQL inserts…to selects…to objects and back to web forms– Many format translations…
Name:
Street:
City:
Zip:
Web Forms Objects RDBMS
17M
D
And the W3C Said…• Let their be simplicity…what if we…allowed
you to XML in the browser…and database
html
headbody
input
labelPerson
FirstName
LastName
Projects
Project
Roles
model
Database
18M
D
A Single Line of SQL
INSERT into MY_TABLE values(XMLPARSE(document '
<MyDocument>
…
</MyDocument>
'))
The equivalent SQL for the eCRV project would have required “shredding”the XML document into over 25 tables (i.e. 25 separate INSERT statements).
19M
D
Enter the Metadata Registry…
• A metadata registry is a central location in the enterprise that the semantics or meaning of data elements is captured and maintained
<DataElement> <Namespace>http://purl.org/dc/elements/1.1</Namespace> <DataElementName>ResourceTitleText</DataElementName> <Label>Title</Label> <Definition>The name given to the resource, usually by the Creator or Publisher.</Definition></DataElement>
Begin with the end in mind…semantic precision.
20M
D
Demos
• NIEM Subset Generator http://niem.gtri.gatech.edu/niemtools/ssgt
• Certificate of Real Estate Value– http://proptax.mdor.state.mn.us:8088/exist/rest//db/crv/forms/submit/index.xhtml
• Contacts Demo– http://proptax.mdor.state.mn.us:8088/exist/rest/db/contacts/index.xhtml
21M
D
22M
D
23M
D
XML Schema (XMLSpy Diagram)
Graphical notation that most non-programmers can learn in 20 minutes.
Dashed linemeans optional
Solid linemeans required
CardinalityDefinition
Sequenceof items
Data Element
24M
D
XForms User Interface (Generated)
Use XSL to transform XML Schema directly into XformsUse the metadata registry to extract element-specific rulesSeparate semantics from constraints
25M
D
Metadata Driven Apps Data Flow
patterns.xmlread-only.xml
DocumentConstraint
Schema.xsd
NIEMMDR
NIEM SubSchemas
.xsd
import
LocalMDRs
My-Form.xhtml
niem2xforms.xsl
wantlist.xml
xforms-global.css
field-widths.css
instance.xml
screen-labels.xmlimport
import
This is a simplified version of the actual workflow.
26M
D
How About Reporting on XML data?
• OK… it is easy to get XML into DB2..
…but how about reporting? How do I do “joins” on XML?
…How do I update XML documents?
…Can I mix SQL and XQuery?
27M
D
DB2 Viper Architecture
• Many benefits of mix and match SQL/Xquery• Use the right tool for the right job
28M
D
29M
D
30M
D
XQuery
• W3C Recommendation 23 January 2007
• Created by experts from SQL industry– IBM, Microsoft, Oracle– IBM played a leadership role
• XQuery is to XML as the SELECT statement is to SQL
31M
D
XQuery Structure
• FLWOR expression (pronounced “flower”)– For – select the sequence of nodes in the tree– Let – temporary variables and lists– Where – restrict results– Order – order results– Return – return the data embedded in a tree
32M
D
Sample XQueryfor $d in db2-fn:xmlcolumn(“depts.depts")//deptnolet $e := db2-fn:xmlcolumn(“emps.emps")//employee[deptno = $d]where count($e) >= 10order by avg($e/salary) descending return<big-dept> { $d, <headcount>{count($e)}</headcount>, <avgsal>{avg($e/salary)}</avgsal> } </big-dept>
Returns a list of all big departments (over 10 employees),their headcount and average salary directly in XML format
33M
D
XSLT and XQuery• Functionally similar languages developed
by the W3C
• Working together to ensure a common approach where appropriate.
• They share the same– data model– type system(XML Schema data types)– function library– Both us XPath 2.0 as a sublanguage
34M
D
What I like About XQuery
• XPath expressions are very short but can still easy to read (c.f. Perl)
• Consistent with XSL, XForms, XML Schema and Schematron (XML rules)
• You can extend XQuery using stored procedures
• Recursion simplifies many tasks
35M
D
But Can I…• Do joins between XML documents…yes
• Extend XQuery…yes
• Create “functions” with XQuery…yes
• Integrate Java functions…yes
36M
D
Joins in XQuery
declare variable $firstName external;<videos featuring="{$firstName}">{ for $v in db2-fn:xmlcolumn(‘videos’)//video, $a in db2-fn:xmlcolumn(‘actors’)//actors/actor where ends-with($a, $firstName) and $v/actorRef = $a/@id order by $v/year return <video year="{$v/year}"> {$v/title} </video>}</videos>
List of videos in one fileList of actors in anotherReturn a list of all videosThat have an actor with agiven first name.
37M
D
XQuery Update• Adds XQuery expressions to:
– Insert– Update– Delete
XML documents
• Still in W3C Working Draft– Last call for Comments is Oct. 2007– Should be full recommendation by 2008
• Available in DB2 9.1 as a stored procedure• Additional support in DB2 9.5
38M
D
Stored Procedures For UpdateCREATE PROCEDURE ECRVADMIN.INCREMENT_COUNTY_CRVID (IN COUNTY VARCHAR(255), OUT SUCESS INT) LANGUAGE SQL BEGIN SET ARG1 = CONCAT( '<updates><update action="compute" col="1" path="//County[Code="' COUNTY "]/NextID/text()">)
CPMCALL DB2XMLFUNCTIONS.XMLUPDATE( ARG1, ‘ select COUNTY from ECRVADMIN.COUNTY where id=0', 'update ECRVADMIN.COUNTY set COUNTY=? where id=0',?,?);
CALL DB2XMLFUNCTIONS.XMLUPDATE( '<updates> <update action="compute" col="1" path="//County[Code=" "]/NextID/text()"> ?+1 </update> </updates>', 'select COUNTY from ECRVADMIN.COUNTY where id=0', 'update ECRVADMIN.COUNTY set COUNTY=? where id=0',?,?);
CALL DB2XMLFUNCTIONS.XMLUPDATE( '<updates> <update action="compute" col="1" path="//County[Code="COUNTY"]/NextID/text()"> ?+1 </update> </updates>', 'select COUNTY from ECRVADMIN.COUNTY where id=0', 'update ECRVADMIN.COUNTY set COUNTY=? where id=0',?,?);
END
39M
D
Sidebar: The Semantic Web
• Note that the semantic web infrastructure is based on graphs, not trees
• Each node in the graph can be a URL• External web sites can be used to create “joins”
based on a URL
Node
Node
Node
Node
Subject ObjectPredicate
Node
Node
40M
D
Strategic Impact
• Consider this:– 50% of Applications are simple CRUD Applications– Create, Read, Update, Delete
• Complex business logic can be stored in an XML-based rules engine such as Schematron
• How will this transform an organization?• How will it allow business units to build and
maintain IT applications?• How will it allow the business to be responsive to
custom needs?
41M
D
Change Where the Line is Drawn
• Shorten the “distance” between the business unit and the IT staff
SME BAs Developers
RequirementsRequirements
SME/BA IT Staff
Graphical Requirements and Specifications
vs.
42M
D
Many Small vs. One Large
It is easier to build a GUI on a small, precise language than a single large language
XFormsXQuery
XPath
XSLT XML Schema
Schematron
Requirements
43M
D
Data Stewardship/Governance
• Importance of the business units role in defining data elements semantics “on-the-wire”
• How to write and maintain enterprise definitions– Concise– Precise– Distinct– Non-circular– Unencumbered
44M
D
Security and Performance
• Security and access control based on URL patterns
• A middle tier still needed to cache web pages that have similar URLs
• Deep REST considerations (don’t regnerate a pages if you don’t have to)
• Stateless web server farms can still be used
45M
D
Modules and Quality Assurance
• XQuery “Modules” allow encapsulation of functions to manipulate complex XML
• Need better unit testing tools for XQuery (similar to JUnit)
• Need support for regression testing frameworks (similar to CruiseControl)
46M
D
Parker Projection
Time
100%
RelativeCode Base
Procedural code (Java, JavaScript, VB, C#, C++)
Declarative code (XHTML, CSS, XSLT, XQuery, XForms)
Source: Jason Parker, Minnesota Department of Revenue, November 2006
47M
D
10 Business Empowerment Steps1. Make data stewardship a business unit imperative2. Start by putting a simple glossary of terms and their
definitions in a database3. Build a simple metadata registry based on a simple
taxonomy4. Build a metadata "shopping cart"5. Build exchange documents using XML Schemas that
import your metadata shopping lists6. Transform XML Schemas into XForms7. Put your XML data in DB2 XML data types (no
shredding!)8. Use XQuery for reports directly into XML and HTML9. Express level-two business rules by selecting XPath
expressions10. Build XML-centric ESB with DB2 databases as sources
48M
D
Next Steps
1. Proof of Technology (POT) – hands on labs
2. Get the evaluation version…start with some small XForms and XQueries
3. Create a pilot project – goal – small middle tier1. Pick a "friendly" business unit
2. Find a medium sized problem
4. Train staff with XPath-centric tools
5. Roll out to the enterprise in stages
49M
D
50M
D
51M
D
Developerworks Article
http://www.ibm.com/developerworks/library/x-xformsniem
52M
D
XForms Wikibookhttp://
en.wikibooks.org/wiki/Xforms
Over 75 working XForms example programs
53M
D
XQuery Book• XQuery
• Priscilla Walmsley
• Tutorial and Reference
• O’Reilly - 2007
• Everything except XQuery update
54M
D
Devx Article
• Semantics and the Evolution of Specialized Languages
http://www.devx.com/semantic/Article/34805
55M
D
The IBM Workplace XForms Video
http://www.ibm.com/developerworks/library/x-xformspointclick.html• One-click generation of XML instance data • Drag-and-drop of complex type elements to create
groups of UI controls that are already bound to data • Drag-and-drop of UI controls from the design palette • Drag-and-drop association of data to UI controls created
from the design palette • Wizard-based generation of data-driven hiding/showing of
groups of UI controls • Wizard-based generation of tables, including add row and
delete row buttons • Wizard-based generation of intelligent row calculations
and column summations
56M
D
MoreResources
• Keywords: XForms, XQuery, PureXML
• This presentation:– http://www.danmccreary.com/presentations/db2ug
57M
D
Useful Links
• DB2 9 Trial– http://www.ibm.com/software/data/db2/v9
• DB2 9 Express-C - Free limited cpu version– http://www.ibm.com/software/data/db2/udb/db2expres
s/download.html
• IBM DeveloperWorks– http://www.ibm.com/developerworks/xml– http://www.ibm.com/developerworks/db2
• IBM DB2 9 PureXML Wiki– http://www.ibm.com/developerworks/wikis/display/db2
xml/Home
58M
D
Thank You!
Dan McCreary
Dan McCreary & Associates
Enterprise Data Architecture and Strategy Consulting
dan@danmccreary.com
http://www.danmccreary.com
Cell: (612) 986-1552
Home: (952) 931-9198