OTM Data Management Using Python - otmsig.com · OTM Data Management Using Python • We show you...
Transcript of OTM Data Management Using Python - otmsig.com · OTM Data Management Using Python • We show you...
<Insert Picture Here>
OTM Data Management Using Python Eric Rosenbloom ([email protected]) Solutions Director, Consulting August 2012
OTM Data Management Using Python
• We show you how to: • Use ClientUtil.py to export/import CSV • Write a Python script that uses ClientUtil.py to automate an
OTM to OTM data replication process using CSV approach • Use ClientUtil.py to export/import db.xml • Use ClientUtil.py to push a GLogXML document to OTM • Time permitting, touch briefly upon using Python to perform
XML data transformations
Target Audience
• This talk is meant primarily for technically inclined individuals responsible for OTM data management
Background on Python
• Python is an interpreted scripting language that comes pre-installed on your OTM application server along with OTM specific utilities like ClientUtil.py
• If you have no prior experience with Python, suggest doing the python tutorial at http://docs.python.org/tutorial/
• Examples presented utilize Python that is pre-installed on an OTM application server. • However, they will work as well when installed on a laptop or
desktop
Diagram: ClientUtil.py in the middle
CSV, db.xml, and GLogXML
• CSV = “Comma Separated Variable”. Standard file format whereby fields are delimited by commas. Supported by Microsoft Excel
• Db.xml is a “database-centric” xml format where there is a close mapping between the xml and the underlying table and columns
• GLogXML is described by the GLogXML.xsd schema. Specifies XML representation for OTM business objects like Item, Location, Release, PlannedShipment, etc
PYTHONPATH environment variable
• On your OTM application server, set PYTHONPATH environment variable to include directory where OTM Python utilities are installed: export PYTHONPATH=$GLOG_HOME/utils/integration/python
Contents of utils/integration/python
Running ClientUtil.py for help • Run ClientUtil.py with no arguments to get help text:
ClientUtil.py to CSV export python ClientUtil.py
-command csvExport -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -tableName ACTIVITY -whereClause "rownum < 2" -localDir . -localFileName myfile.csv -excludePublic N
Myfile.csv contents ACTIVITY ACTIVITY_GID,ACTIVITY_XID,ACTIVITY_NAME,DOMAIN_NAME,INSERT_USER,INSERT_D
ATE,UPDATE_USER,UPDATE_DATE EXEC SQL ALTER SESSION SET NLS_DATE_FORMAT = 'YYYYMMDDHH24MISS' "RECEIVE","RECEIVE","RECEIVING FREIGHT","PUBLIC","DBA.ADMIN","20110130185513",,
ClientUtil.py to CSV export: Ex #2 python ClientUtil.py -command csvExport -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -tableName LOCATION -whereClause "1=1" -localDir . -localFileName location.csv -excludePublic Y
Export a subset of columns • The csvExport command exports all columns in the
table • If you want a subset of the columns in the table, use
the csvQuery command instead of the csvExport command
CSV query example python ClientUtil.py -command csvQuery -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -sqlQuery "SELECT location_gid FROM location" -localDir . -localFileName myfile.csv
Myfile.csv contents SELECT location_gid FROM location LOCATION_GID "PHILLY" "NYC" ,
CSV Import Example: data content
LOCATION LOCATION_GID,LOCATION_XID,COUNTRY_CODE3
_GID,DOMAIN_NAME GUEST.SIG2012,SIG2012,USA,GUEST ,
CSV import example: command-line python ClientUtil.py -command csvImport -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -localDir . -localFileName new_location.csv -xvalidate Y -csvUtilCommand ii
CSV import example: xml output <?xml version="1.0" encoding="UTF-8"?> <CSVUtilServletHelper> <CSVUtil> <Command>ii</Command> <DataDir>/home/oracle/otm62/temp/upload/</DataDir> <DataFileName>Transmission1338186382454.csv</DataFileName> <ExcludePublic>true</ExcludePublic> <ProcessCSV> <xvalidate>true</xvalidate> <DatabaseGlobalName>CONSAPP06.US.ORACLE.COM</DatabaseGlobalName> <TableName>LOCATION</TableName> <ColumnList>LOCATION_GID,LOCATION_XID,COUNTRY_CODE3_GID,DOMAIN_NAME</ColumnList> <sqlString>insert into LOCATION (
LOCATION_GID,LOCATION_XID,COUNTRY_CODE3_GID,DOMAIN_NAME) values (?,?,?,?)</sqlString> <ProcessCount>1</ProcessCount> <ErrorCount>0</ErrorCount> <SkipCount>0</SkipCount> </ProcessCSV> </CSVUtil> </CSVUtilServletHelper>
Data Replication Script
• Now let’s write our own “copytabs.py” Python script to use ClientUtil to facilitate replicating data from OTM instance A to OTM instance B
copytabs.py: command line
copytabs.py: includes
copytabs.py: main
copytabs.py: Main class
copytabs method (all in list)
copytab method (copy one table)
Rate Data Replication Script
• Now let’s write a shell script that uses copytabs.py to copy rate data from one OTM instance to another
copyrates.sh
ClientUtil.py to single table db.xml export
python ClientUtil.py -command xmlQuery -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -sqlQuery "select * from activity" -rootName ACTIVITY -localDir .
-localFileName activity.db.xml
single table db.xml export
ClientUtil.py to multi-table db.xml export
python ClientUtil.py -command xmlQuery -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -sqlQuery "select location.*, (cursor (select location_role_profile.* from location_role_profile where location_role_profile.location_gid = location.location_gid)) as location_role_profile from location where domain_name = ‘GUEST’ " -rootName LOCATION -localDir . -localFileName location.db.xml
multi-table db.xml export
ClientUtil.py db.xml import
python ClientUtil.py -command xmlImport -hostname consapp07 -username GUEST.ADMIN -password CHANGEME -transactionCode IU -localDir . -localFileName location.db.xml
db.xml import
Sample “Location” GLogXML document
ClientUtil.py: Push GLogXML document
python ClientUtil.py -command sendXMLFileViaHttpPost -url http://consapp07/GC3/glog.integration.servlet.WMServlet -fileName Location.xml
Sample Command Line and Response
Login to OTM
Query Location Manager
Python for XML Data Transformations
• Performing XML data transformations in Python is relatively straight-forward
• Helper methods in commonTools.py • Sample script SampleLocationTransform.py
SampleLocationFlatFile.loc
SampleLocationTransform.py
Summary
• You have learned • How to get started with Python • About CSV, db.xml, and GLogXML • About the various Python tools that come installed with OTM • How to use ClientUtil.py to export/import CSV and db.xml • Intro to writing your own Python scripts • Use ClientUtil.py to
• export/import db.xml • Push GLogXML to OTM
• Introduction to using Python for XML data transformations
If you want to learn more…
• Reference the OTM Data Management Guide:
• http://download.oracle.com/docs/cd/E20111_01/otm/acrobat/datamanagement.pdf
If you want to learn more…