ABSOLUTELY TYPICAL - THE WHOLE STORY ON TYPES AND HOW THEY POWER PL/SQL INTEROPERABILITY

Lucas Jellema, AMIS

On Types and Objects, Collections and OOUKOUG, December 2011, Birmingham, UK

OVERVIEW

• History of Types• Areas of use• Type• True OO• Collections [of Types]• Table and Multiset• Interaction with the outside world

– XML, JSON– Web Services

STANDARD TYPES IN ORACLE DATABASE• Standard as in ‘installed out of the box’

– Number, Char, Date– Varchar2– RAW, LONG– SDO_GEOMETRY, …– BLOB, CLOB, TIMESTAMP, INTERVAL– XMLType – DICOM (Medical Records)

THE DATABASE SUFFIX

• 12c• 11g• 10g• 9i• 8i• 8.0

– That is 8.o (as in oh, not as in zero!)• And: 8 really is• OO in Oracle turned out to mean

– Define user defined types to use as SQL and PL/SQL data structure programming constructs• that have some OO characteristics as well• that can be used to define table columns for

oooo

AREAS OF USE FOR TYPES

• Structured programming in PL/SQL• Interaction SQL and PL/SQL

– Elegant, performing, organized• Interaction with the outside world

– Java, SOA Suite, other consumers• OO-like storage of nested type structures• Do OO design and application development

EXAMPLE OF TYPE DEFINITION

• Types are manipulated through DDL statements: – CREATE OR REPLACE TYPE

– Object Types have properties (attributes)– Object Types are similar to PL/SQL records– Object Types live in Data Dictionary as primary

database objects

Type as Object

PERSON_T

INSTANTIATING AN OBJECT

• A variable based on a type (as object) needs to be instantiated before it can be assigned or read

• An instance of a type == an object

FUNCTION ACTING ON TYPE

CALLING FUNCTION WITH TYPE PARAMETER FROM SQL• Types can be instantiated from within SQL

statements and passed to PL/SQL functions

– SQL*Plus and other tools typically know how to render types in query results

SELECT OBJECT TYPE FROM SQL

SCALAR SUBQUERY

• Inline query that returns a single result– One row, one column

select ename, job, sal, (select count(empno) from emp s where s.mgr= e1.empno)from emp e1

select ename, sal from empwhere sal between (select avg(sal) from emp where job = 'SALESMAN') and (select avg(sal) from emp where job = 'ANALYST')

Employees with their job and sal and the number of staff they manage

Employees who earn between the average salary for salesmen and analysts

WORKAROUND TO RETRIEVE MULTIPLE VALUES: CONCATENATION OF VALUES

select ename, job, sal, substr(sal_aggs, 1, instr(sal_aggs,'x') -1) avg_sal_in_job, substr(sal_aggs, instr(sal_aggs,'x') +1) max_sal_in_jobfrom ( select ename , job , sal , ( select avg(sal)||'x'||max(sal) from emp where job = e1.job ) sal_aggs from emp e1 )order by job, sal desc

MUCH BETTER WORK AROUND: ADT

select ename, job, sal, emps.sal_aggs.value1 avg_sal_in_job, emps.sal_aggs.value2 max_sal_in_jobfrom ( select ename , job , sal

, ( select multivalue_return_type

( avg(sal), max(sal), null) from emp where job = e1.job ) sal_aggs from emp e1 ) empsorder by job, sal desc

create typemultivalue_return_typeas object( value1 number(20), value2 number(20), value3 number(20))

TYPES AND THE DBA

• Before Oracle Database 11g – altering type definitions was not easy– Drop, Alter or Replace of a Type could only be done if no

references existed to the type from other types or tables

– That usually meant all types associated with an application were dropped and recreated – to alter one!

• Starting in 11g the DROP and REPLACE/ALTER TYPE operations can specify FORCE to ignore dependencies– Helps to overcome frowning, type-averse DBAs!

• Note: Data Dictionary Views are available to tell on types

• Types are editionable objects in terms of 11gR2 Edition Based Redefinition and can evolve gracefully

COMPLEX TYPES

• Attribute of a Type (as Object) can be based on an ADT aka UDT: Object or Collection– Limitation: Self-reference is not allowed

Type as Object

PERSON_T

Type as Object

SOCIAL_PROFILE_T

SOCIAL PROFILES STORED IN TABLES• Two options for storing social profiles:

– The normal, explicit relational way:

– The nested tableapproach:

INSTANTIATING NESTED OBJECT IN SQL• Instantiate ‘look up object’ using Scalar

Subquery

• Instantiate ‘look up object’ from (outer) joined columns

COLLECTIONS IN ORACLE

• Oracle supports in-memory collections of type instances– Can be of standard type (number, varchar2,

date,…) or complex User Defined Type (aka ADT)• Three kinds of collections

– Associative Array (PL/SQL only) – {key,value} pairs, very similar to Map in Java and Hashtable in C#

– VARRAY (the variable array of fixed size)– Nested Table

• These collections are similar, yet quite distinct– Definition and instantiation– Similar operations (count, first, next, delete, …)– Sparse vs. Dense– SQL Interaction support

CREATING AND WORKING WITH AN ASSOCIATIVE ARRAY (PL/SQL ONLY)

CREATING AND WORKING WITH A NESTED TABLE

COLLECTION OPERATIONS

• Exists – to check if element with/at index exists

• Extend (only for Nested Table)• Count• First• Last• Next• Prior• Delete• Trim

COMPLEX TYPES

Type as Object

PERSON_T

Type as Object

SOCIAL_PROFILE_T

Type as Object

EMAIL_ADDRESS_T

Type as Table of email_address_t

EMAIL_ADDRESS_TBL

CONSTRUCTING A COMPLEX OBJECT IN PL/SQL

CONSTRUCTING A COMPLEX OBJECT IN PL/SQL

TRUE OO: SUBTYPES AND INHERITANCE• Types can be created as subtypes of another

Type• A subtype has multiple identities:

– EMPLOYEE_T is both PERSON_T and EMPLOYEE_T– Anything that can handle a PERSON_T can also

handle an EMPLOYEE_T and a CUSTOMER_T

Type as Object

PERSON_T

Type as Object

CUSTOMER_T

Type as Object

EMPLOYEE_T

CREATION OF TYPE HIERARCHY

Type as Object

EMPLOYEE_T

CREATE INSTANCES OF SUBTYPE IN PL/SQL

OO OPERATORS IN TYPE HIERARCHY• TREAT (AS type) – downcast an instance to a

specific subtype – to access specialized subtype aspects

• IS OF ( [ONLY] type) – to identify type of an instance

MEMBER FUNCTIONS

• A type can have operations– Specification and Implementation (in the Body)– Acting on the members – public and private – of

the object– Possibly overriding methods of the super-type

• Two types of operations– Constructor– Normal member functions

Type PERSON_T as Object

<public methods>

Type BodyPERSON_T

<implementation of private and

public methods>

CONSTRUCTOR FUNCTIONS

• A type is instantiated through a call to one of its constructor methods

• A type can have multiple, overloaded user defined constructor functions– In addition to the

default system defined constructor that simply expects input for every public member

MULTIPLE USER DEFINED CONSTRUCTORS

MEMBER FUNCTIONS

• Types can have member functions – similar to functions in a package– Just like package can act on package state

(globals), member functions act on object (type instance)

MEMBER FUNCTION DISPLAY_LABEL

OVERRIDING MEMBER FUNCTIONS

• Sub types can override member functions from super– When the function is invoked on a PERSON_T

and PERSON_T happens to be a CUSTOMER_T, then the function as defined on CUSTOMER_T is executed

– Overriding subtype member function can invoke the overridden super typer functionType PERSON_T as Object

Type CUSTOMER_T as ObjectType EMPLOYEE_T as Object

EXAMPLE OF OVERRIDING MEMBER FUNCTION

Type PERSON_T as Object

Type CUSTOMER_T as Object

MAP AND ORDER MEMBER FUNCTIONSORTING COLLECTIONS

• Map Member Function translates an object to a scalar data type that Oracle knows how to compare and sort– Varchar2, Number, Date or even an UDT with

scalar attributes (or a Map function of its own)

• Note: Map member function is also used for equality comparison– In this case John Doe and Bill Doe are seen as equal!

MAP MEMBER FUNCTION IN ACTION - SORTING IN SQL

• Map function dictates:– Women come first, then sort by birth date

(young before old)

ORDER MEMBER FUNCTION USED TO ORDER BY IN SQL• Order member function to compare objects

– returns -1 when self comes before other (when sorted), 1 when self comes after and 0 when they draw

ORDER MEMBER FUNCTION IN ACTION

USER DEFINED AGGREGATION FUNCTIONS IN SQL

• The Oracle Data Cartridge offers User Extensibility framework for the database

• Data Cartridge provides: – user defined aggregate functions– domain indexes and functions

• used for Oracle Text and Oracle Spatial

– Pipelined Table Functions – the interface approach• Dynamic handling of AnyData objects

select avg(hiredate) from emp

ORA-00932: inconsistent datatypes: expected NUMBER got DATE

IMPLEMENTING USER DEFINED AGGREGATE• Implement an object with the following

operations:– ODCIAggregateInitialize – ODCIAggregateIterate – ODCIAggregateTerminate – ODCIAggregateMerge

• Optional, to enable parallelism

create type DateAvgImplas object ( count number , sumdates NUMBER , function ODCIAggregateInitialize , function ODCIAggregateIterate , function ODCIAggregateTerminate , function ODCIAggregateMerge )

IMPLEMENTING USER DEFINED AGGREGATE• Register new aggregate function implemented by

the object

• Use the new function in queries

• “Analytic Function Friendly”

function dateavg (input date) return datePARALLEL_ENABLEAGGREGATE USING DATEAVGIMPL

select DateAvg(hiredate) from emp

FUNCTION CHAINING

• Dot notation (navigation) is used to call a function on the result of a function (on the result…)– if the result of the nested functions are types

BULK COLLECT TO PREVENT MULTIPLE ROW-BY-ROW PL/SQL TO SQL ACCESS

• Traditional cursor based approaches for fetching data from the database using SQL resulted in – Unnecessary SQL/PLSQL context switches

• By fetching multiple rows at once – the number of (expensive) switches can be reduced

• Several approaches exist – all rely on Collections– Bulk Collect– Select Collection

• Note: as of Oracle 10g the ‘normal’ cursor-for-loop also implicitly does fetching in bulk

BULK COLLECT EXAMPLE

CREATE A COLLECTION IN SQL – COLLECT OPERATOR• COLLECT is an aggregation function (like

MAX, SUM, COUNT) that aggregates into nested tables– Group by can be used to create ‘clusters’

SELECT COLLECT INTO PL/SQL VARIABLE

MULTISET TURNING A (SUB) QUERY RESULT SET ON THE FLY INTO A NESTED TABLE

• MULTISET can be used inside SQL Queries to convert a query result set into a collection (nested table)– On the fly (in the middle of a query)

MULTISET EXAMPLE

MULTISET OPERATORS ACTING ON NESTED TABLES IN SQL AND/OR PL/SQL

• C1 = C2– Comparison based on named type, cardinality and

comparison of individual elements• C1 <> C2 or C1 != C2

– Inverse of equality test• C1 IN C2 returns true if C1 is one of the

collections in C2 which is a collection of collections

• POWERMULTISET C1 generates all non-empty sub-multisets from collection C1– All possible combinations created from the

elements in C1, produced as collections of the same type as C1

• C1 IS EMPTY returns true if C1 is empty• CARDINALITY C1 returns the number of elements

in C1

MULTISET OPERATORS ACTING ON NESTED TABLES IN SQL AND PL/SQL• C1 SUBMULTISET C2 evaluates to true when the

members of C1 are all found in C2• X MEMBER OF C1 evaluates to true when X is

found as a member of collection C1• C1 MULTISET UNION [DISTINCT] C2 – combining

two collections– Use DISTINCT to eliminate duplicate elements

• C1 MULTISET INTERSECT [DISTINCT] C2 – produces collection of elements that occur in both C1 and C2

• C1 MULTISET EXCEPT [DISTINCT] C2 – produces the collection of elements that appear in C1 and not in C2– Use ALL to allow (reduced number of )duplicates

• C1 IS A SET – tests if the collection C1 only has unique members (no duplicates)

COULD BE ANYTHING

• The Any… Types are like generic placeholders for data that is only known at run time– The contents of Any… instances is accessible

through introspection

• AnyData – some thing, could be any thing (Object)

• AnyType – holds the definition of some thing• Like AnyData without the data – only meta data

• AnySet – some collection of things (all of the same type)– can be used for interface parameters to

communicate self-descriptive sets of data– Used in Pipelined Table Functions created through

Data Cartridge to return dynamically determined data structures

CREATING AND PROCESSING A TABLE OF THINGS – STEP 1

CREATING AND PROCESSING A TABLE OF THINGS – STEP 2

TABLEProcess collection as query result

TABLE FUNCTIONS

• In queries: select from collections as if they were a table using the TABLE operator

JUST BECAUSE WE CAN - POINTLESS COMBINATION OF COLLECT AND TABLE

SORTING COLLECTIONSLEVERAGE SQL TO DO THE ORDER BY

Build up collection of peoplein variable l_people

EXTREME ENCAPSULATION

• View on top of table(collection)– With instead of trigger handling DML

– Data queried from NUMBERS is produced in PL/SQL – no underlying table involved

– Data manipulated in NUMBERS is processed in memory

EXTREME ENCAPSULATION

FOR GENERATING ROWS FOR SPECIAL DOMAINS OF ALLOWABLE VALUES

• Have TABLE operate on an in-line created collection

• To save on repeated– select value from dual UNION ALL select value

…

with countries as( select column_value country from table( string_table('Belgium','France' ,'Egypt','Italy' ) ))select countryfrom countries

with gender_values as( select column_value gender from table( string_table('MALE', 'FEMALE',‘UNKNOWN' )))select genderfrom gender_values

PIPING ROWS USING COLLECTIONS AND THE ‘TABLE FUNCTION’ • A Table Function normally returns the entire

collection at once• However: it can also return collection

elements one at a time– That is called PIPELINED

• As soon as the function pipes a single element, processing that element can continue

• It’s bit like the /*+ FIRST_ROWS */ Hint

THE SLOW ALPHABET…

create or replace function alphabetreturn letter_tableis l_alphabet letter_table:= letter_table();begin l_alphabet.extend(26); for i in 1..26 loop l_alphabet(i):= chr(64+i); dbms_lock.sleep(0.1); end loop; return l_alphabet;end alphabet;

Note: it is not just 2.79 seconds for all rows to appear: it takes that long for any row to appear

THE PIPELINED SLOW ALPHABET…

create or replace function alphabet return letter_tablepipelinedis l_alphabet letter_table:= letter_table();begin l_alphabet.extend(26); for i in 1..26 loop l_alphabet(i):= chr(64+i); dbms_lock.sleep(0.1); pipe row (l_alphabet(i)); end loop; return;end alphabet;

Note: total processing time is just as long as before; however, the first row appears after less than 0.5 sec!

MORE PERFORMANCE REQUIRES PARALLEL

MORE PERFORMANCE REQUIRES PARALLEL

DO NOT DO IT…ON YOUR OWN

• Parallel means: multiple resources contributing to a task at the same time

• Introduce parallellism into your database application– parallel query and DML– coordinated jobs (dbms_job, dbms_scheduler)– 11g: dbms_parallel_execute– Pipelined & Parallel table functions

PANCAKE PARTY

BETTER PERFORMING PANCAKE PARTY

PIPELINED PANCAKE PARTY: BEST PERFORMANCE

PIPELINED PANCAKE PARTY

• Parallel Pipelined: multiple different resources working on different components of the task in [potentially] different tiers– From batch processing to unit processing => pass

the unit on for further processing as soon as part of the task is done – to leverage resources (across tiers) in parallel

• Instead of baking the entire stack first and only then eating it…– … start eating as soon as the first one is done– Even the last guest ends earlier than without

pipelining provided he eats more than one pancake• (he eats his first pancake when it is done, not when

the stack is done)

– The first eater is done much sooner • first rows/all rows ≈ first pancake/all pancakes

PANCAKE PARTY IN PL/SQL

PANCAKE PARTY IN PL/SQL

PANCAKE PARTY PIPELINED IN PL/SQL

PIPELINED PANCAKE BA-EAT-KING

INTRODUCING XML

• Structured ASCII• Esperanto• Cross technology – interoperability• Tools & frameworks for common operations

– Parsing, validation, construction, transformation and querying

• Standards: – XSD – design and contract– XPath - querying– XSLT - transformation– XQuery – querying and manipulation

WHAT IS XML?

eXtensible Markup Language – for exchange of data

Plain text / ASCII files Structured messages

Human readable More important: machine interpreted

Meaning defined with tags Elements and attributes<?xml version="1.0" encoding="UTF-8"?><root element>

<element1 attributeName="value"> content</element1><element2 attributeName="value" /><element3> <element4>content</element4></element3>

</root element>

TYPE TO HOLD XML DATA IN ORACLE

XMLType

CONSTRUCTING XML IN DATABASE APPLICATIONS• Many ways to produce XML content in Oracle

– XMLType – from text, types, cursor, bfile contents, …

– SQL/XML (XMLAgg, XMLElement, XMLForest, …)– DBMS_XMLGEN– SYS_XMLGEN and SYS_XMLAGG– DBURIType and XDBURIType– XML SQL Utility (XSU) – command line, Java,

PL/SQL– fn:doc and fn:collection (replacing ora:view)

XMLType

CONVERTING TYPE STRUCTURE TO XMLTYPE• XMLType can be instantiated based on a

single complex User Defined Type instance– Not (directly) on a nested table

CONVERTING NESTED TABLE TO XMLTYPE

CONVERT XMLTYPE TO NESTED USER DEFINED TYPE• XMLType supports conversion directly in a –

potentially nested – user defined type– Provided element names match names of type,

subtypes and attributes

XML CAN BE SOMEWHAT HEAVY

• Size of XML documents compared to the actual data content of those documents is ‘XXL’

• Complex nested structure that requires parsing

• Turning XML into native data structures and vice versa is expensive (Marshalling/Unmarshalling )

• XML Programming facilities not available on all platforms and in all languages– Objective C, JavaScript, …

• The search for alternativesis on: structured, contract, standardized, facilities,lighter weight!

JavaScript Object Notation

Lightweight data-interchange format

Name : Value

Name : Value,Name : Value,Name : Value

Name : Value,Name : Value,Name : Value

Name : Value,Name : Value,Name : Value

Name : Value,Name : Value,Name : Value, ,

{"ACCOUNTING" : { "EMPLOYEES" : [ {"ENAME" : "KING", "JOB" : "PRESIDENT", "SAL" : 5000 }, {"ENAME" : "MILLER", "JOB" : "CLERK", "SAL" : 1300 }] }}

<department name="ACCOUNTING"> <employees> <employee> <ename>KING</ename> <job>PRESIDENT</job> <sal>5000</sal> </employee> <employee> <ename>MILLER</ename> <job>CLERK</job> <sal>1300</sal> </employee> </employees></department>

PL/JSON EXAMPLE – CREATING AND QUERYING JSON

MANIPULATION OF JSON OBJECT

FROM TYPES TO XML TO JSON

FROM JSON TO XML TO TYPES

PUBLISH PACKAGE AS HTTP-BASED API USING DBMS_EPG• Hide database protocol

– Not its physical location nor the schema, and user authentication

• HTTP communication is truly cross technology– Browser, Java, .Net, JavaScript & RIA clients, …

• Is also possibly remote, across networks, firewalls etc. – easier than normal JDBC connections

• Can publish in various formats– Text, HTML, CSV, JSON, XML, RSS

• Use case:cross-technology, internal no WS*/ESB

http

SIMPLE DBMS_EPG EXAMPLE

sys

scott

scott

RESTFUL WEBSERVICES

• Simple, stateless, light weight, http-based message exchange protocol– Use standard get, put, post, delete operations– Use URL to pass resource reference

• Small, easy-to-understand messages• Typically XML or JSON based• Not using SOAP, or the WS*-stack• Often used directly from User Interface –

(AJAX enabled) JavaScript

RESTFUL ARCHITECTURE

RESTful PL/SQL APIexposed through dbms_epg

httphttp

RESTFUL RESOURCE NAVIGATION

PUBLISH REST-FUL HTTP API – DIRECTLY ON TOP OF THE RDBMS

PUBLISH REST-FUL HTTP API – DIRECTLY ON TOP OF THE RDBMS

JSON RESPONSE – WROUGHT FROM TYPES AND COLLECTIONS

TURNING ANY OBJECT INTO JSON

CONSUMING REST-FUL SERVICES

• Invoking a REST-ful service is simply making an HTTP request to some URL

• Can be done through:– utl_http– httpuritype– Stored Java

• Result can be processed as just a string or as XML, JSON or other format– Libraries (PL/JSON) can help with special formats

SOAP WEB SERVICES

• All messages (input and output) are XML• The message consists of two parts inside an

envelope (a SOAP XML wrapper)– The header with meta-data– The body with the contents to be handled by or

returned by the service• The WebService Definition Language (WSDL)

document describes the service• An XML Schema Document (XSD) describes

the structure of the XML messages– XSD is like an ERD or Table Design

ORACLE RDBMS 11G – NATIVE DATABASE WEB SERVICES• Database Schema can be published

through native database web services– Each package corresponds with a WSDL

• With an operation for each Procedure or Function• Auto-mapping from PL/SQL input and output

parameters to WSDL (XSD) Types

– WSDL and XSD are dynamically generated– A generic Query Service (across schema) is

exposed• To execute user defined queries• With support for pagination and some formatting

WS/SOAP

– http and https is supported– Limited control over WSDL & XSD

– Use case: internal, cross technology, WS enabled client, no ESB or Application Server available

STUFF INC

• Challenge: publish SOAP Web Services to expose data services based on an Oracle Database– Read data and Manipulate data

• Use as decoupled an approach as possible– Java developers do not

touch SQL or PL/SQL– Database developers are not

aware of web service context

• See white paper on OTN http://www.oracle.com/technetwork/articles/soa/jellema-esb-pattern-1385306.html for details

Chat/IM XMPP Server

HTTP

JDBC

Other(Email, FTP/File,

XMPP/Chat)

SOA Suite

Oracle Service

Bus

DB

AQ

JMS

EJB

FileFTP

SDO

WS

http

PL/SQL package

Table

AQ

Native DB WebService

EPG

View

JEE Server Database

JAX-WS

ADF BC/SDO WS

EJB/JPA

Email ServerFile/FTP Server

UMS

XMLDB

XMLTypes

XMLXML & XSD

JSON/ CSV

Ref Cursor

Types & CollJPublisher

WS

utl_file, BFILE,

URITYPE

JMS Queue

XMLRelational/Oracle Type

JMS

Adapters

Pojo

PL/SQL MUSIC API EXPRESSED IN TYPES AND COLLECTIONS

PackageType

Package

XMLT

ype

Type

SOAP

Web

Serv

ice

JAX-WS

Java Class

Java ClassOracle JDBC

XMLType exchange

Java Class

Mediator

The ESB Architecture pattern:

DecoupleValidate, Enrich, Transform, Route, …

FOR JAVA DEVELOPERS BENEFIT: WRAP TYPE IN XMLTYPE

SOAP

Web

Serv

ice

JAX-WS

Java Class

Java ClassOracle JDBC

XMLType exchange

Java Class

Mediator Package

XMLT

ype

Type

String

JDBC

CLO

B

TO ACCOMMODATE FRAMEWORKS THAT DO NOT UNDERSTAND XMLTYPE

USING THE OSB OR SOA SUITE DATABASE ADAPTER – USER DEFINED TYPES ARE FINE

• Database adapter interacting with Type based PL/SQL API– Derives XSD from input and output parameters– Allows for perfect decoupling between PL/SQL

on one side of the fence and Web Service/XML on the other

PackageType

SOAP

Web

Serv

ice

Proxy Service

Validate Enrich Transform Route & Operate

XML

HTTPSOAP

Business Service

Database Adapter

User Defined

Type

JDBC

PackageType

SOAP

Web

Serv

ice

Proxy Service

Validate Enrich Transform Route & Operate

XML

HTTPSOAP

Business Service

Database Adapter

User Defined

Type

JDBC

JUKEBOX SERVICE - COMPLETE

SUMMARY

• Long live types!• Structured programming in PL/SQL

– Multiple input and output parameters, step by step result constructions

• Integration between SQL and PL/SQL– Scalar Subquery, Bulk Collect and Collect

Aggregator– Table and Multiset operators– Pipelined Table Functions

• Integrating with the external world– Type turning XML and/or JSON– Native Database Web Services– Database Adapter in SOA Suite and Service Bus