<Insert Picture Here>

OTM Data Management Using Python Eric Rosenbloom (eric.rosenbloom@oracle.com) Solutions Director, Consulting August 2012

OTM Data Management Using Python

• We show you how to: • Use ClientUtil.py to export/import CSV • Write a Python script that uses ClientUtil.py to automate an

OTM to OTM data replication process using CSV approach • Use ClientUtil.py to export/import db.xml • Use ClientUtil.py to push a GLogXML document to OTM • Time permitting, touch briefly upon using Python to perform

XML data transformations

Target Audience

• This talk is meant primarily for technically inclined individuals responsible for OTM data management

Background on Python

• Python is an interpreted scripting language that comes pre-installed on your OTM application server along with OTM specific utilities like ClientUtil.py

• If you have no prior experience with Python, suggest doing the python tutorial at http://docs.python.org/tutorial/

• Examples presented utilize Python that is pre-installed on an OTM application server. • However, they will work as well when installed on a laptop or

desktop

Diagram: ClientUtil.py in the middle

CSV, db.xml, and GLogXML

• CSV = “Comma Separated Variable”. Standard file format whereby fields are delimited by commas. Supported by Microsoft Excel

• Db.xml is a “database-centric” xml format where there is a close mapping between the xml and the underlying table and columns

• GLogXML is described by the GLogXML.xsd schema. Specifies XML representation for OTM business objects like Item, Location, Release, PlannedShipment, etc

PYTHONPATH environment variable

• On your OTM application server, set PYTHONPATH environment variable to include directory where OTM Python utilities are installed: export PYTHONPATH=$GLOG_HOME/utils/integration/python

Contents of utils/integration/python

Running ClientUtil.py for help • Run ClientUtil.py with no arguments to get help text:

ClientUtil.py to CSV export python ClientUtil.py

-command csvExport -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -tableName ACTIVITY -whereClause "rownum < 2" -localDir . -localFileName myfile.csv -excludePublic N

Myfile.csv contents ACTIVITY ACTIVITY_GID,ACTIVITY_XID,ACTIVITY_NAME,DOMAIN_NAME,INSERT_USER,INSERT_D

ATE,UPDATE_USER,UPDATE_DATE EXEC SQL ALTER SESSION SET NLS_DATE_FORMAT = 'YYYYMMDDHH24MISS' "RECEIVE","RECEIVE","RECEIVING FREIGHT","PUBLIC","DBA.ADMIN","20110130185513",,

ClientUtil.py to CSV export: Ex #2 python ClientUtil.py -command csvExport -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -tableName LOCATION -whereClause "1=1" -localDir . -localFileName location.csv -excludePublic Y

Export a subset of columns • The csvExport command exports all columns in the

table • If you want a subset of the columns in the table, use

the csvQuery command instead of the csvExport command

CSV query example python ClientUtil.py -command csvQuery -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -sqlQuery "SELECT location_gid FROM location" -localDir . -localFileName myfile.csv

Myfile.csv contents SELECT location_gid FROM location LOCATION_GID "PHILLY" "NYC" ,

CSV Import Example: data content

LOCATION LOCATION_GID,LOCATION_XID,COUNTRY_CODE3

_GID,DOMAIN_NAME GUEST.SIG2012,SIG2012,USA,GUEST ,

CSV import example: command-line python ClientUtil.py -command csvImport -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -localDir . -localFileName new_location.csv -xvalidate Y -csvUtilCommand ii

CSV import example: xml output <?xml version="1.0" encoding="UTF-8"?> <CSVUtilServletHelper> <CSVUtil> <Command>ii</Command> <DataDir>/home/oracle/otm62/temp/upload/</DataDir> <DataFileName>Transmission1338186382454.csv</DataFileName> <ExcludePublic>true</ExcludePublic> <ProcessCSV> <xvalidate>true</xvalidate> <DatabaseGlobalName>CONSAPP06.US.ORACLE.COM</DatabaseGlobalName> <TableName>LOCATION</TableName> <ColumnList>LOCATION_GID,LOCATION_XID,COUNTRY_CODE3_GID,DOMAIN_NAME</ColumnList> <sqlString>insert into LOCATION (

LOCATION_GID,LOCATION_XID,COUNTRY_CODE3_GID,DOMAIN_NAME) values (?,?,?,?)</sqlString> <ProcessCount>1</ProcessCount> <ErrorCount>0</ErrorCount> <SkipCount>0</SkipCount> </ProcessCSV> </CSVUtil> </CSVUtilServletHelper>

Data Replication Script

• Now let’s write our own “copytabs.py” Python script to use ClientUtil to facilitate replicating data from OTM instance A to OTM instance B

copytabs.py: command line

copytabs.py: includes

copytabs.py: main

copytabs.py: Main class

copytabs method (all in list)

copytab method (copy one table)

Rate Data Replication Script

• Now let’s write a shell script that uses copytabs.py to copy rate data from one OTM instance to another

copyrates.sh

ClientUtil.py to single table db.xml export

python ClientUtil.py -command xmlQuery -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -sqlQuery "select * from activity" -rootName ACTIVITY -localDir .

-localFileName activity.db.xml

single table db.xml export

ClientUtil.py to multi-table db.xml export

python ClientUtil.py -command xmlQuery -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -sqlQuery "select location.*, (cursor (select location_role_profile.* from location_role_profile where location_role_profile.location_gid = location.location_gid)) as location_role_profile from location where domain_name = ‘GUEST’ " -rootName LOCATION -localDir . -localFileName location.db.xml

multi-table db.xml export

ClientUtil.py db.xml import

python ClientUtil.py -command xmlImport -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -transactionCode IU -localDir . -localFileName location.db.xml

db.xml import

Sample “Location” GLogXML document

ClientUtil.py: Push GLogXML document

python ClientUtil.py -command sendXMLFileViaHttpPost -url http://consapp07/GC3/glog.integration.servlet.WMServlet -fileName Location.xml

Sample Command Line and Response

Login to OTM

Query Location Manager

Python for XML Data Transformations

• Performing XML data transformations in Python is relatively straight-forward

• Helper methods in commonTools.py • Sample script SampleLocationTransform.py

SampleLocationFlatFile.loc

SampleLocationTransform.py

Summary

• You have learned • How to get started with Python • About CSV, db.xml, and GLogXML • About the various Python tools that come installed with OTM • How to use ClientUtil.py to export/import CSV and db.xml • Intro to writing your own Python scripts • Use ClientUtil.py to

• export/import db.xml • Push GLogXML to OTM

• Introduction to using Python for XML data transformations

If you want to learn more…

• Reference the OTM Data Management Guide:

• http://download.oracle.com/docs/cd/E20111_01/otm/acrobat/datamanagement.pdf

If you want to learn more…