Post on 11-Dec-2015
TH Technology
Karen Cannellkcannell@thtechnology.com
http://www.thtechnology.com
How Do I Load Data …Let Me Count The WaysAPEX Data Loading Options
How Do I Load Data …
TH Technology
How Do I Load Data? : Agenda
Oracle Data Loading OptionsDBAs - Developers - End Users
APEX Data Loading OptionsFile Upload
Data Load Wizard2
XLS Upload – APEX Listener
WebSheets – Copy/Paste, Add Row
TH Technology
How Do I Load Data …
About Me …
Karen Cannell ~ Consultant, TH Technology
SW Engineer 25+ years, Oracle since 1994. Building APEX apps for government, medical, engineering industries.Leveraging the Oracle 10g,11g (now 12c) suite of tools
Beginning Application Express, APress, 2011Agile Application Express, APress, 2011
Editor, ODTUG Technical JournalVolunteer to author ODTUG Journal Articles!
Using APEX since the HTMLDB beginning
How Do I Load Data …
TH Technology
About You … (Audience Background)
New to APEX?
APEX Experience?
Previous Tools?
APEX Training?
Version 2.0? 3.1 ? 3.2 ? 4.1?
How Do I Load Data …
TH Technology
How DO I Load Data into Oracle?
Load Data Into Oracle
7,340,000 hits
SQL Loader
Oracle Database Utilities
Load Excel Data into Oracle
How Do I Load Data …
TH Technology
How DO I Load Data into Oracle?
Load Data Into APEX
317,000 hits
Import Excel Data …
APEX Data Loader
Load Text File into APEX
How Do I Load Data …
TH Technology
How DO I Load Data into Oracle?
Load XLS Data Into APEX
1,830,000 hits
Import Excel File …
APEX Data Loader
Load Excel Data …
Yikes!
TH Technology
How Do I Load Data …
Load Data into Oracle - Options
What Are the Options?
Commercial Solutions - Excel-DB, Quickload
Warehouse Builder, ETL Tools
Oracle Data Integratorhttp://www.oracle.com/technetwork/middleware/data-integrator/overview/index.html
Oracle Database UtilitiesOracle Data Pump
SQL Loader
External Tables
TH Technology
How Do I Load Data …
Load Data into Oracle, cont’d
ODBC Connections
IDEs w/built-In Data Loaders
Custom Code:Java
PL/SQL Using UTL_FILE
Scripting from XLS Columns
Java and PL/SQL
Perl
Other
TH Technology
How Do I Load Data …
Let’s Focus …
TH Technology
How Do I Load Data …
Load Data into APEX
Import Excel Data into Oracle …
APEX Data Loader
“Load Excel Data”
95% Say XLS but Really are CSV
The Apex Data Loader is a java based application that can bulk process the insert, update and delete on all object data into and
build queries to extract data out of salesforce.com using the Apex Web Services
(SOAP) API.
TH Technology
How Do I Load Data …
Our Problem
“Load Data into Oracle”
40% Fixed-Format Text Files
60% Excel Spreadsheets
Several XLS Formats
200+ Files Monthly
Locations All Over the World
Some Networked, Some Not
Varying Volume of Data
TH Technology
How Do I Load Data …
Which to Choose?
It Depends Application
Source
Target ( Table?)
Amount of Data
How Often ? 1 or Many
Audience End Users
Resources $$
Time $$
TH Technology
How Do I Load Data …
Which Make Sense for APEX ?
Oracle Database Utilities
PL/SQL IDE
No Additional Licenses
PL/SQL Code
No Complex Setup
Developer-Friendly
End User Friendly
TH Technology
How Do I Load Data …
Oracle Database Utilities
Oracle Data PumpOracle DB to Oracle DB
Faster than EXP and IMPEXPDP 2x faster than EXP
IMPDP 15-45x faster than IMP
Jobs can be Restarted
Supports Network Import and ExportLoad one instance form another
Remote export
TH Technology
How Do I Load Data …
Oracle Database Utilities, cont’d
SQL Loader
External files into Tables
Variety of formats
Filtering
Multiple Table Load in One Load Session
Conventional
Direct Path
External Table Load
TH Technology
How Do I Load Data …
Oracle Database Utilities, cont’d
External Tables
External files into Tables
Variety of Formats
Preprocessing Capability
External Tables to PullRPT Files
CSV Files - That We Can Get
TH Technology
How Do I Load Data …
Oracle Database – Database Link
Don’t Forget the Database Link!
SELECT … FROM table@db_link
Simple, Direct
Requires Communication
Ask!
TH Technology
How Do I Load Data …
Use your IDE
SQL Developer
PL/SQL Developer
Toad
Others …
Best for One-Time Load
TH Technology
How Do I Load Data …
SQL Developer – Data Load Option
Import Data …
Wizard
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
Import File Formats
.XLS / XLSX
.CSV
.TSV (tab)
.DSV ( SQL
Server)
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
Import Wiz
Detects• File type• Delimiter• Skip Row
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
INSERT or
Generate
INSERT
Script)
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
Select
Columns
to
Import
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
Map
Source
to
Table
Columns
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
Verify
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
Finished!
XLS and XLSX* File Load in Very Few Clicks!
*XLSX in SQL Developer 3.1
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
What About XLSX?• SQL Dev 3.0 – No• SQL Developer 3.1 – Yes
XLS/XLSX File Load in Very Few Clicks!
TH Technology
How Do I Load Data …
SQL Developer – Import Data …
ProsOne Shot Data Load
Delimiter (, TAB, others)
Flexible Columns
Flexible Mapping
XLS File
Can be Faster than APEX Data Load
Wizard
ConsTable Must Be Defined
Cannot Integrate into App
Tedious to Repeat
IDE Specific
One Shot Data Load
Not for End Users
TH Technology
How Do I Load Data …
APEX Data Load/Unload Utility
APEX From the Beginning
Easy Load of Spreadsheet or XML Data
New or Existing Table
Upload File Or Copy/Paste
Delimiter Option
Column by Column Selection
TH Technology
How Do I Load Data …
APEX Data Load/Unload Utility
TH Technology
How Do I Load Data …
APEX Data Load/Unload Utility
1st Row Column Heading
TH Technology
How Do I Load Data …
APEX Data Load/Unload Utility
TH Technology
How Do I Load Data …
APEX Data Load/Unload Utility
Specify PK and Trigger
Finish
TH Technology
How Do I Load Data …
APEX Data Load Utility
Pros
Simple: Point-Click-Done
Flexible Format
Spreadsheet Convenience
Creates Table
Copy/Paste Option
Cons
Many Steps
Single Table
No XLS Upload (Must Copy/Paste or Save as CSV)
One-Time
Not for End Users
TH Technology
How Do I Load Data …
What About the End User?
APEX Standard File Browse + Custom Code
Data Load Wizard WizardAPEX Manual Data Load Wizard End User-ized
APEX Web Sheet Data GridCopy/Paste, Data Entry
APEX Listener XLS UploadFile Browse + XLS2COLLECTION
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Most Common (Still)
File Browse Item
Process (to Staging)
Process To Destination Table(s)
MANY MANY OPTIONS …
All Require Custom Code
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
File Browse Item
WWV_FLOW_FILES – Original Way
BLOB column – New Option w APEX 4+
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
WWV_FLOW_FILES – Traditional Way
File uploads to APEX Table WWV_FLOW_FILES
Move Records to a Custom Table (i.e. Staging Table)
Clean up!Files accumulate in APEX_FILES Tablespace
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
BLOB Column – New w/ APEX 4
File Stored in BLOB Column of Table Specified in Automatic Row Processing (DML) Process
Column Specified in Item Source
Table Must HaveBLOB
Filename
MIME Type
CHARSET
Saves an Archiving Step
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
BLOB Column – New w/ APEX 4
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
WWV_FLOW_FILES
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
LOCAL_UPLOAD_FILES
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Parse / Load Process
Read Blob into local BLOB variable
Read Records from Blob
Load Records into Destination Objects
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Reusable Parse/ Load Process
File Browse ItemLoad File to WWV_FLOW_FILE
Generic Parse File PackageGeneric File Upload Utility
Generic Store to APEX Collection
Process Code per Use/Table/Format
Custom Code, But Less of It
TH Technology
How Do I Load Data …
Generic Parse File Utility Package
Parse File to Collection
Load Collection to Specified TableColumns Must Match, or Adjust Code
Reusable Code, Pages for Any Upload File/Table
Saves One Part of Custom Code
Custom Processing from Stage Table Required
Custom Code, But Less of It
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Parse File Process
Read Blob into local BLOB variable BEGIN
SELECT blob_content INTO l_blob FROM wwv_flow_files WHERE name=p_file_name;EXCEPTION WHEN NO_DATA_FOUND THEN raise_application_error(-20000, 'File not found, id='||p_file_name);END;
Or BLOB column
from your BLOB table
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Parse File Process
Read Records from Blob Variable TYPE varchar2_t IS TABLE OF VARCHAR2(32767) INDEX BY binary_integer; l_records VARCHAR2_T; … get_records(l_blob,l_records); … -- Initialize the APEX collection apex_collection.create_or_truncate_collection( p_collection_name);
TH Technology
How Do I Load Data …
PROCEDURE get_records(p_blob IN blob,p_records OUT varchar2_t) IS l_sep VARCHAR2(2) := chr(13)||chr(10); l_last INTEGER; l_current INTEGER;BEGIN IF (NVL(DBMS_LOB.INSTR(p_blob, utl_raw.cast_to_raw(l_sep),1,1),0)=0) THEN l_sep := chr(10); END IF; l_last := 1; LOOP l_current := dbms_lob.INSTR( p_blob, utl_raw.cast_to_raw(l_record_separator), l_last, 1 ); EXIT WHEN (nvl(l_current,0) = 0); p_records(p_records.count+1) := utl_raw.cast_to_varchar2( DBMS_LOB.SUBSTR(p_blob,l_current-l_last,l_last)); l_last := l_current+length(l_record_separator); END LOOP;END get_records;
TH Technology
How Do I Load Data …
LOOP l_current := dbms_lob.INSTR( p_blob, utl_raw.cast_to_raw(l_record_separator), l_last, 1 ); EXIT WHEN (nvl(l_current,0) = 0); p_records(p_records.count+1) := utl_raw.cast_to_varchar2( DBMS_LOB.SUBSTR(p_blob,l_current-l_last,l_last)); l_last := l_current+length(l_record_separator); END LOOP;
Loop to read records from the BLOB, line by line(separator to separator)
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Parse File Process: Read from Blob,
Load Collection FOR i IN 2..l_records.count LOOP csv_to_array(l_records(i),l_record); l_seq_id := apex_collection.add_member(p_collection_name,'dummy'); FOR i IN 1..l_record.count LOOP apex_collection.update_member_attribute( p_collection_name=> p_collection_name, p_seq => l_seq_id, p_attr_number => i, p_attr_value => l_record(i)); END LOOP;
END LOOP;
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Parse File Process: Read Blob, Load Collection
Insert from Collection to Table …
l_ddl := ‘INSERT INTO '||p_table_name||' '|| ‘SELECT '||v(p_columns_item)||' '|| ‘FROM apex_collections '|| ‘WHERE seq_id >= 1 and collection_name='''|| p_collection_name||'''';
EXECUTE IMMEDIATE l_ddl; …
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Parse File Process: Read Blob, Load Collection, Insert,
Remove File -- Cleanup file! DELETE FROM wwv_flow_files WHERE name=p_file_name;
TH Technology
How Do I Load Data …
APEX File Browse + Custom Code
Process to Destination Table(s)
INSERT /* +append */ … SELECT … FROM staging
MERGE statement
Whatever is Required for Your Business
Custom Code
TH Technology
How Do I Load Data …
File Browse & Custom Code
ProsCustom Code
Custom Solution
ConsCustom Code
Resources
Maintenance
One-of Solution
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
The SQL Workshop Data Load Utility End-User-ized
Page Type
Creates a Shared Component
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Wizard to Create A Data Load Wizard
Create Page Data Loading
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Define Table Lookups
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Add TransformsTo Upper/Lower, R/L/Trim, Single Whitespace
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Page/Region/Breadcrumb Names
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Define Tabs (if any)
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Define Branches
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Confirmation
TH Technology
How Do I Load Data …
Data Load Wizard - Result
Working Upload for our USERS table
TH Technology
How Do I Load Data …
Data Load Wizard
End User Sees:Data Load Progression
Chance to Select Row to Include/Ignore
Failed Rows at the End
Chance to Back Out Load?
Chance to Reload Clean?
TH Technology
How Do I Load Data …
Data Load Wizard
Upload OR Copy/Paste
Data Limit ==
Specify Delimiter
Specify Enclosed By
Specify Currency, Group Separator, Decimal Characters
TH Technology
How Do I Load Data …
Data Load Wizard - Options
TH Technology
How Do I Load Data …
Data Load Wizard – Pro’s/Con’s
Pros
Flexible Form
Delimiter Option
1st Row Heading Opt
Currency – Group Separator – Decimal
Mapping Built In
Repeatable for End User
Reusable for Developer
Cons
Too Much for End User??
No XLS
Copy/Paste Size Limit
Upload Size Limit ??
Single Transform per Column
Home
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Is It Foolproof?
No:
Ex: RPT file …. What does this means?
Too Many Columns?
Too Few?
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Can I trick it into taking an XLS?
No:
TH Technology
How Do I Load Data …
Data Load Wizard
Can I Make it “Better”?
Try:
Default/Eliminate All Options
Eliminate Load/Unload Column Mapping
Catch/Warn of File Type, # Columns Errors
Load the Collection Behind the Schemes
TH Technology
How Do I Load Data …
Data Load Wizard – Behind the Scenes
Usual Expected ItemsFILE_BROWSE
TH Technology
How Do I Load Data …
Data Load Wizard – P 2
Parse Uploaded DataSingle Process of Type Parse Uploaded Data
List of Parameters
Fires upon Next
TH Technology
How Do I Load Data …
TH Technology
How Do I Load Data …
Data Load Wizard – P 3
Process Uploaded DataSingle Process of Type Process Uploaded Data
List of Parameters
Fires upon Next
TH Technology
How Do I Load Data …
Data Load Wizard – P 3
Session InfoCollections
Character Columns
Numeric Columns
Date Columns
TH Technology
How Do I Load Data …
TH Technology
How Do I Load Data …
Data Load Wizard Wizard
Pros
In-App
Straightforward
Reusable
No Custom Code for Data Load into DB
Cons
Too Many End User Steps
Still Need Custom Processing Code for Staging to Tables
IF get P1 Finish by Loading Collection,
have a Two-Step File Upload for End
Users!
TH Technology
How Do I Load Data …
APEX Listener XLS Upload
File Browse ItemLoad File to WWV_FLOW_FILE
Parsing FileCustom Code per Use/Table/Format
Generic File Upload Utility
Custom Process to Destination Tables
TH Technology
How Do I Load Data …
Parse File to Collection
Loads Collection to Specified TableTable Exist or Not
Options
Reusable for Any Upload File/Table
Saves One Part of Custom Code
Custom Processing from Stage Table Required
APEX Listener XLS Upload
TH Technology
How Do I Load Data …
APEX Listener XLS Upload
Demo
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
Must Deploy via APEX ListenerCurrent Release 1.1.3…
Allows Upload of XLS File into a Collection
Best Reference: (from RMOUG 2010)
http://krisrice.blogspot.com/2010/02/another-apex-listener-ea-more-knobs-to.html
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
Install APEX Listener
Deploy
Edit apex_config.xml
Add Configuration Choices:
<entry key="apex.excel2collection">true</entry>
<entry key="apex.excel2collection.onecollection">true</entry>
<entry key="apex.excel2collection.name">EXCEL_COLLECTION</entry>
<entry key="apex.excel2collection.useSheetName">true</entry>
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
Configuration Options:
Turn Feature On:
<entry key="apex.excel2collection">true</entry>
One Collection per Worksheet:
<entry key= "apex.excel2collection.onecollection">true|false</entry>
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
Configuration Options:
Specify Name for All Collections:
<entry key=
"apex.useSheetName">true|false</entry>
false == sequential sheet name
true == uses sheet name
Specify Name of Collection for All Collections:
<entry key=
"apex.excel2collection.collection.name">MY_COLLECTION</entry>
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
File Browse ItemWWV_FLOW_FILES option
Page Item Button Name or REQUEST = XLS2COLLECTION
Report Region:SELECT * FROM apex_collectionsWHERE collection_name = ‘MY_COLLECTION’
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
“Files ending with .xls or .xlsx will be processed.”
BUT
.xls process
Macro-Enabled .xls process
.XLSX do NOT Process as of APEX Listener 1.1.3.243.11.40
.XLSX will Process as of v2
Coming Soon!
TH Technology
How Do I Load Data …
XLSX Upload via APEX Listener
TH Technology
How Do I Load Data …
XLSX Upload via APEX Listener
SEVERE: service exception:org.apache.poi.poifs.filesystem.OfficeXmlFileException: The supplied data appears to be in the Office 2007+ XML. You are calling the part of POI that deals with OLE2 Office Documents. You need to call a different part of POI to process this data (eg XSSF instead of HSSF)
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
Formulas Translate Literally
Ex: IF(E2="","",VLOOKUP(E2,$AE$4:$AF$218,2,FALSE))
TH Technology
How Do I Load Data …
XLS Upload via APEX Listener
STILL Need Custom Code to Process Data from Collection to Database Table(s)
TH Technology
How Do I Load Data …
APEX Listener XLS Upload
Pros
XLS File – No Need to Save As CSV
Straightforward
Reusable
Less Custom Code
Cons
APEX Listener Deployment
Embedded Formulas
No XLSX - May Need a Save As Step
Will Have XLSX in V2
TH Technology
How Do I Load Data …
APEX Websheets
Data Grid Allows Copy/Paste
Spreadsheet-Like Edits
Data Grid w/in Page
Custom Code Required to Transfer Data to Main Schema Tables
TH Technology
How Do I Load Data …
Editable Data in Websheet
TH Technology
How Do I Load Data …
APEX Websheets
Demo
TH Technology
How Do I Load Data …
Pull Data From Websheet into App
Look at APEX$_WS Tables:APEX$_WS_ROW – the data
APEX$_WS_HISTORY – iff you care about who changed what when
Need to process from SQL Dev table to “real” tables
TH Technology
How Do I Load Data …
APEX WebSheets
Pros
Copy/Paste
Spreadsheet-Like Entry, Edits
End User Can Build
Cons
Custom Code to Create Websheet, Data Grid?
Custom Code Required to Pull Data into Schema Objects
Custom Code to Maintain Worksheet / Data Grid
More Code Than Other Solutions ?
TH Technology
How Do I Load Data …
Our Solution
Problem
RPT Files
XLS files
Different Formats
200+, Monthly
Users Cannot “Save as CSV”
Must Support Edits
Varying Volume
Limited Network
Solution
Tabular Forms - 4
Macro to Save as RPT
External TablesGenerated RPT
LIS Output RPT Files
Upload File Process (to be retired)
Custom ProcessingRPT – LI Systems
RPT – Spreadsheet Templates
TH Technology
How Do I Load Data …
APEX Data Load Options
TH Technology
How Do I Load Data …
Nice to Have … Future?
1-Step Data Load WizardCopy/Paste OR
Upload XLSX
Easiest for Non-Networked, Low(er) Volume End Users
TH Technology
How Do I Load Data …
Some Last Thoughts …
Forms Create/Edit One Row
Tabular FormsEnter Many Rows
Java Solution.JSP to load XLS directly
Presume this means XLSX too!
TH Technology
How Do I Load Data …
How Do I Load Data ?
APEX-based Solution
Weigh RequirementsSource Data
End User Capabilities
Resources
Time
No One-Size-Fits-All
Apply Common Sense
TH Technology
How Do I Load Data …
TH Technology
How Do I Load Data …
Common Sense Test …
Two men play 5 games of checkers. Each wins the same # of games. There are no ties.
How can this be?
They are not playing each other.
How Do I Load Data …
TH Technology
References
Upload File into APEX
oraexplorer.com – Ittichai
http://oraexplorer.com/2007/11/apex-to-upload-text-file-and-write-into/
More at avdeo.com In Oracle Milieu
http://avdeo.com/2008/05/21/uploading-excel-sheet-using-oracle-application-express-apex/
How Do I Load Data …
TH Technology
References
Oracle Data Integratorhttp://www.oracle.com/technetwork/middleware/data-integrator/overview/index.html
Oracle Database UtilitiesSQL Loader – External Tableshttp://www.oracle.com/technetwork/database/enterprise-edition/index-093639.htm
l
SQL Developerhttp://www.oracle.com/technetwork/developer-tools/sql-developer/overview/index.html
How Do I Load Data …
TH Technology
References
APEX Listener XLS Upload – Kris Rice Blog
http://krisrice.blogspot.com/2010/02/another-apex-listener-ea-more-knobs-to.html
A Java-and-PL/SQL based XLS to CSV Optionhttp://www.dynamicpsp.com/!go?ln=nfaq&faq=FAQ9
TH Technologykcannell@thtechnology.com
How Do I Load Data?
Let Me Count The Ways
Questions ?
Please fill out
the
evaluations!
TH Technologykcannell@thtechnology.com
Thank You
Please fill out
the
evaluations!
How Do I Load Data?
Let Me Count The Ways