Xi321 Ds Reference En

SAP BusinessObjects Data Services Reference Guide SAP BusinessObjects Data Services XI 3.2 SP1 (12.2.1)

2010-11-23

2010 SAP AG. All rights reserved.SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAPBusiness ByDesign, and other SAP products and services mentioned herein as well as their respective

Copyright

logos are trademarks or registered trademarks of SAP AG in Germany and other countries. BusinessObjects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, WebIntelligence, Xcelsius, and other Business Objects products and services mentioned herein as wellas their respective logos are trademarks or registered trademarks of Business Objects S.A. in theUnited States and in other countries. Business Objects is an SAP company.All other product andservice names mentioned are the trademarks of their respective companies. Data contained in thisdocument serves informational purposes only. National product specifications may vary.These materialsare subject to change without notice. These materials are provided by SAP AG and its affiliatedcompanies ("SAP Group") for informational purposes only, without representation or warranty of anykind, and SAP Group shall not be liable for errors or omissions with respect to the materials. Theonly warranties for SAP Group products and services are those that are set forth in the expresswarranty statements accompanying such products and services, if any. Nothing herein should beconstrued as constituting an additional warranty.

2010-11-23

Contents

Introduction...........................................................................................................................13Chapter 1

Welcome to SAP BusinessObjects Data Services.................................................................131.1

Welcome...............................................................................................................................131.1.1

Documentation set for SAP BusinessObjects Data Services.................................................131.1.2

Accessing documentation......................................................................................................161.1.3

SAP BusinessObjects information resources.........................................................................171.1.4

Overview of this guide............................................................................................................181.2

About this guide.....................................................................................................................181.2.1

Who should read this guide....................................................................................................181.2.2

Objects..................................................................................................................................21Chapter 2

Characteristics of objects......................................................................................................212.1

Object classes ......................................................................................................................212.1.1

Object options, properties, and attributes..............................................................................232.1.2

Descriptions of objects .........................................................................................................242.2

Annotation.............................................................................................................................272.2.1

Batch Job..............................................................................................................................282.2.2

Catch ....................................................................................................................................372.2.3

COBOL copybook file format ................................................................................................412.2.4

Conditional ............................................................................................................................482.2.5

Data flow ..............................................................................................................................492.2.6

Datastore...............................................................................................................................522.2.7

Document............................................................................................................................1022.2.8

DTD.....................................................................................................................................1032.2.9

Excel workbook format ........................................................................................................1142.2.10

File format............................................................................................................................1212.2.11

Function...............................................................................................................................1342.2.12

Log......................................................................................................................................1352.2.13

Message function................................................................................................................1422.2.14

Outbound message..............................................................................................................1422.2.15

Project.................................................................................................................................1432.2.16

Query transform...................................................................................................................1432.2.17

2010-11-233

Real-time job........................................................................................................................1442.2.18

Script...................................................................................................................................1482.2.19

Source.................................................................................................................................1482.2.20

Table....................................................................................................................................1582.2.21

Target..................................................................................................................................1622.2.22

Target Writer migrated from Data Quality ............................................................................2022.2.23

Template table.....................................................................................................................2022.2.24

Transform............................................................................................................................2052.2.25

Try.......................................................................................................................................2062.2.26

While loop............................................................................................................................2062.2.27

Work flow............................................................................................................................2072.2.28

XML file...............................................................................................................................2092.2.29

XML message......................................................................................................................2132.2.30

XML schema........................................................................................................................2152.2.31

XML template......................................................................................................................2312.2.32

Smart editor........................................................................................................................233Chapter 3

Accessing the smart editor...................................................................................................2333.1

Smart editor options............................................................................................................2343.2

Smart editor toolbar.............................................................................................................2343.2.1

Editor Library pane...............................................................................................................2353.2.2

Editor pane...........................................................................................................................2373.2.3

To browse for a function......................................................................................................2403.3

To search for a function........................................................................................................2413.4

Data Types..........................................................................................................................243Chapter 4

Descriptions of data types...................................................................................................2434.1

date.....................................................................................................................................2444.1.1

datetime...............................................................................................................................2464.1.2

decimal................................................................................................................................2474.1.3

double .................................................................................................................................2474.1.4

int........................................................................................................................................2484.1.5

interval.................................................................................................................................2484.1.6

Limitations for long and blob.................................................................................................2484.1.7

numeric................................................................................................................................2524.1.8

real.......................................................................................................................................2534.1.9

time......................................................................................................................................2534.1.10

timestamp............................................................................................................................2544.1.11

varchar.................................................................................................................................2554.1.12

Data type processing...........................................................................................................2574.2

2010-11-234

Contents

Date arithmetic....................................................................................................................2574.2.1

Type conversion...................................................................................................................2584.2.2

Transforms..........................................................................................................................283Chapter 5

Operation codes..................................................................................................................2835.1

Descriptions of transforms...................................................................................................2845.2

Data Integrator transforms...................................................................................................2875.2.1

Data Quality transforms.......................................................................................................3515.2.2

Platform transforms..............................................................................................................5145.2.3

Functions and Procedures..................................................................................................557Chapter 6

About functions....................................................................................................................5576.1

Functions compared with transforms....................................................................................5576.1.1

Operation of a function.........................................................................................................5576.1.2

Arithmetic in date functions..................................................................................................5586.1.3

Including functions in expressions........................................................................................5586.1.4

Built-in functions...................................................................................................................5606.1.5

Descriptions of built-in functions..........................................................................................5646.2

abs.......................................................................................................................................5736.2.1

add_months.........................................................................................................................5746.2.2

ascii.....................................................................................................................................5756.2.3

avg.......................................................................................................................................5766.2.4

base64_decode...................................................................................................................5766.2.5

base64_encode...................................................................................................................5776.2.6

begin_trex_transaction.........................................................................................................5786.2.7

cast......................................................................................................................................5796.2.8

chr.......................................................................................................................................5816.2.9

ceil.......................................................................................................................................5826.2.10

commit_trex_transaction......................................................................................................5836.2.11

concat_date_time ................................................................................................................5846.2.12

count ...................................................................................................................................5846.2.13

count_distinct.......................................................................................................................5856.2.14

create_trex_cube.................................................................................................................5866.2.15

current_configuration ..........................................................................................................5876.2.16

current_system_configuration..............................................................................................5886.2.17

dataflow_name ....................................................................................................................5886.2.18

datastore_field_value...........................................................................................................5886.2.19

date_diff...............................................................................................................................5896.2.20

date_part .............................................................................................................................5906.2.21

day_in_month.......................................................................................................................5916.2.22

2010-11-235

Contents

day_in_week........................................................................................................................5926.2.23

day_in_year..........................................................................................................................5936.2.24

db_type................................................................................................................................5946.2.25

db_version...........................................................................................................................5956.2.26

db_database_name..............................................................................................................5976.2.27

db_owner.............................................................................................................................5986.2.28

decode.................................................................................................................................5986.2.29

delete_trex_cube .................................................................................................................6006.2.30

double_metaphone...............................................................................................................6016.2.31

exec.....................................................................................................................................6026.2.32

extract_from_xml..................................................................................................................6066.2.33

file_exists ............................................................................................................................6086.2.34

fiscal_day.............................................................................................................................6096.2.35

floor.....................................................................................................................................6096.2.36

gen_row_num_by_group......................................................................................................6106.2.37

gen_row_num .....................................................................................................................6126.2.38

get_domain_description.......................................................................................................6136.2.39

get_env................................................................................................................................6146.2.40

get_error_filename ..............................................................................................................6156.2.41

get_file_attribute..................................................................................................................6156.2.42

get_monitor_filename ..........................................................................................................6166.2.43

get_trace_filename ..............................................................................................................6176.2.44

greatest...............................................................................................................................6176.2.45

host_name ..........................................................................................................................6196.2.46

ifthenelse.............................................................................................................................6196.2.47

index....................................................................................................................................6206.2.48

init_cap................................................................................................................................6216.2.49

interval_to_char....................................................................................................................6226.2.50

is_group_changed................................................................................................................6236.2.51

is_set_env............................................................................................................................6246.2.52

is_valid_date........................................................................................................................6256.2.53

is_valid_datetime..................................................................................................................6266.2.54

is_valid_decimal...................................................................................................................6286.2.55

is_valid_double.....................................................................................................................6296.2.56

is_valid_int...........................................................................................................................6306.2.57

is_valid_real.........................................................................................................................6316.2.58

is_valid_time........................................................................................................................6316.2.59

isempty ...............................................................................................................................6326.2.60

isweekend ...........................................................................................................................6336.2.61

job_name ............................................................................................................................6346.2.62

Job_Run_ID.........................................................................................................................6356.2.63

2010-11-236

Contents

julian....................................................................................................................................6356.2.64

julian_to_date.......................................................................................................................6366.2.65

key_generation.....................................................................................................................6366.2.66

last_date .............................................................................................................................6376.2.67

least.....................................................................................................................................6386.2.68

length ..................................................................................................................................6406.2.69

literal....................................................................................................................................6406.2.70

ln..........................................................................................................................................6426.2.71

load_to_xml..........................................................................................................................6436.2.72

log........................................................................................................................................6456.2.73

long_to_varchar....................................................................................................................6456.2.74

lookup .................................................................................................................................6466.2.75

lookup_ext...........................................................................................................................6506.2.76

lookup_seq..........................................................................................................................6566.2.77

lower ...................................................................................................................................6606.2.78

lpad .....................................................................................................................................6616.2.79

lpad_ext...............................................................................................................................6616.2.80

ltrim .....................................................................................................................................6636.2.81

ltrim_blanks .........................................................................................................................6646.2.82

ltrim_blanks_ext ..................................................................................................................6656.2.83

mail_to.................................................................................................................................6666.2.84

match_pattern......................................................................................................................6686.2.85

match_regex .......................................................................................................................6716.2.86

match_simple.......................................................................................................................6786.2.87

max......................................................................................................................................6806.2.88

min.......................................................................................................................................6816.2.89

mod.....................................................................................................................................6816.2.90

month..................................................................................................................................6826.2.91

num_to_interval....................................................................................................................6836.2.92

nvl........................................................................................................................................6846.2.93

power...................................................................................................................................6856.2.94

previous_row_value..............................................................................................................6866.2.95

print ....................................................................................................................................6876.2.96

pushdown_sql .....................................................................................................................6886.2.97

quarter.................................................................................................................................6906.2.98

raise_exception ...................................................................................................................6916.2.99

raise_exception_ext ............................................................................................................6916.2.100

rand.....................................................................................................................................6926.2.101

rand_ext...............................................................................................................................6936.2.102

replace_substr ....................................................................................................................6946.2.103

replace_substr_ext...............................................................................................................6956.2.104

2010-11-237

Contents

repository_name .................................................................................................................6976.2.105

rollback_trex_transaction......................................................................................................6976.2.106

round...................................................................................................................................6986.2.107

rpad.....................................................................................................................................6996.2.108

rpad_ext...............................................................................................................................7006.2.109

rtrim ....................................................................................................................................7026.2.110

rtrim_blanks ........................................................................................................................7036.2.111

rtrim_blanks_ext ..................................................................................................................7046.2.112

sap_openhub_processchain_execute...................................................................................7046.2.113

sap_openhub_set_read_status.............................................................................................7086.2.114

search_replace.....................................................................................................................7096.2.115

set_env ...............................................................................................................................7146.2.116

sleep ...................................................................................................................................7156.2.117

soundex...............................................................................................................................7156.2.118

sql .......................................................................................................................................7166.2.119

sqrt......................................................................................................................................7186.2.120

smtp_to................................................................................................................................7196.2.121

substr..................................................................................................................................7226.2.122

sum......................................................................................................................................7246.2.123

sysdate................................................................................................................................7256.2.124

system_user_name .............................................................................................................7266.2.125

systime................................................................................................................................7266.2.126

table_attribute .....................................................................................................................7276.2.127

to_char.................................................................................................................................7286.2.128

to_date................................................................................................................................7316.2.129

to_decimal ..........................................................................................................................7326.2.130

to_decimal_ext ....................................................................................................................7336.2.131

total_rows............................................................................................................................7346.2.132

trunc....................................................................................................................................7356.2.133

truncate_table......................................................................................................................7366.2.134

upper...................................................................................................................................7376.2.135

varchar_to_long....................................................................................................................7386.2.136

wait_for_file..........................................................................................................................7386.2.137

week_in_month....................................................................................................................7406.2.138

week_in_year.......................................................................................................................7416.2.139

WL_GetKeyValue ...............................................................................................................7426.2.140

word....................................................................................................................................7436.2.141

word_ext .............................................................................................................................7446.2.142

workflow_name ...................................................................................................................7456.2.143

year......................................................................................................................................7466.2.144

About procedures................................................................................................................7466.3

2010-11-238

Contents

Overview..............................................................................................................................7476.3.1

Requirements.......................................................................................................................7476.3.2

Creating stored procedures in a database............................................................................7486.3.3

Importing metadata for stored procedures ...........................................................................7526.3.4

Structure of a stored procedure...........................................................................................7536.3.5

Calling stored procedures....................................................................................................7546.3.6

Checking execution status ..................................................................................................7596.3.7

Scripting Language.............................................................................................................761Chapter 7

To use the scripting language...............................................................................................7617.1

Language syntax..................................................................................................................7617.2

Syntax for statements in scripts...........................................................................................7617.2.1

Syntax for column and table references in expressions........................................................7627.2.2

Strings.................................................................................................................................7637.2.3

Variables..............................................................................................................................7647.2.4

Variable interpolation............................................................................................................7657.2.5

Functions and stored procedures.........................................................................................7657.2.6

Operators............................................................................................................................7667.2.7

NULL values........................................................................................................................7687.2.8

Debugging and Validation.....................................................................................................7717.2.9

Keywords.............................................................................................................................7737.2.10

Sample scripts.....................................................................................................................7757.3

Square function....................................................................................................................7757.3.1

RepeatString function..........................................................................................................7767.3.2

Metadata in Repository Tables and Views..........................................................................779Chapter 8

Auditing metadata................................................................................................................7798.1

AL_AUDIT............................................................................................................................7798.1.1

AL_AUDIT_INFO..................................................................................................................7808.1.2

Imported metadata...............................................................................................................7828.2

AL_INDEX............................................................................................................................7828.2.1

AL_PCOLUMN....................................................................................................................7838.2.2

AL_PKEY.............................................................................................................................7838.2.3

ALVW_COLUMNATTR........................................................................................................7848.2.4

ALVW_COLUMNINFO........................................................................................................7858.2.5

ALVW_FKREL......................................................................................................................7868.2.6

ALVW_MAPPING................................................................................................................7878.2.7

ALVW_TABLEATTR.............................................................................................................7958.2.8

ALVW_TABLEINFO.............................................................................................................7968.2.9

Internal metadata..................................................................................................................7968.3

2010-11-239

Contents

AL_LANG............................................................................................................................7978.3.1

AL_LANGXMLTEXT.............................................................................................................7988.3.2

AL_ATTR.............................................................................................................................7988.3.3

AL_SETOPTIONS................................................................................................................7998.3.4

AL_USAGE..........................................................................................................................8008.3.5

ALVW_FUNCINFO..............................................................................................................8038.3.6

ALVW_PARENT_CHILD......................................................................................................8048.3.7

Metadata Integrator tables...................................................................................................8058.4

AL_CMS_BV.......................................................................................................................8068.4.1

AL_CMS_BV_FIELDS..........................................................................................................8078.4.2

AL_CMS_REPORTS............................................................................................................8088.4.3

AL_CMS_REPORTUSAGE..................................................................................................8098.4.4

AL_CMS_FOLDER..............................................................................................................8108.4.5

AL_CMS_UNV....................................................................................................................8118.4.6

AL_CMS_UNV_OBJ...........................................................................................................8118.4.7

Operational metadata...........................................................................................................8128.5

AL_HISTORY.......................................................................................................................8128.5.1

ALVW_FLOW_STAT............................................................................................................8138.5.2

Locales and Multi-byte Functionality..................................................................................815Chapter 9

Locale support.....................................................................................................................8159.1

Locale selection...................................................................................................................8179.1.1

Code page support..............................................................................................................8199.1.2

Guidelines for setting locales...............................................................................................8219.1.3

Multi-byte support................................................................................................................8269.2

Multi-byte string functions....................................................................................................8269.2.1

Numeric data types: assigning constant values....................................................................8269.2.2

Byte Order Mark characters.................................................................................................8279.2.3

Round-trip conversion..........................................................................................................8289.2.4

Column sizing.......................................................................................................................8289.2.5

Limitations of multi-byte support..........................................................................................8299.3

Definitions............................................................................................................................8299.4

Supported locales and encodings.........................................................................................8319.5

Data Quality Fields..............................................................................................................835Appendix A

Data Quality fields................................................................................................................835A.1

Content types......................................................................................................................835A.1.1

Associate output fields.........................................................................................................837A.2

Country ID fields..................................................................................................................837A.3

Input Fields...........................................................................................................................838A.3.1

2010-11-2310

Contents

Output fields........................................................................................................................838A.3.2

Data Cleanse fields..............................................................................................................839A.4

Input fields...........................................................................................................................839A.4.1

Output fields........................................................................................................................841A.4.2

Geocoder fields...................................................................................................................849A.5

Input fields...........................................................................................................................849A.5.1

Output fields........................................................................................................................850A.5.2

Global Address Cleanse fields.............................................................................................854A.6

Field category columns in Output tab...................................................................................854A.6.1

Input fields...........................................................................................................................857A.6.2

Output Fields.......................................................................................................................860A.6.3

Global Suggestion Lists fields..............................................................................................873A.7

Input fields...........................................................................................................................873A.7.1

Output fields........................................................................................................................874A.7.2

Match fields.........................................................................................................................877A.8

Match transform output fields..............................................................................................877A.8.1

USA Regulatory Address Cleanse fields..............................................................................882A.9

Field category columns in Output tab...................................................................................882A.9.1

Input fields...........................................................................................................................885A.9.2

Output fields........................................................................................................................887A.9.3

Python.................................................................................................................................909Appendix B

Python.................................................................................................................................909B.1

About Python ......................................................................................................................909B.1.1

Create an expression with the Python Expression editor......................................................912B.1.2

Built-in objects.....................................................................................................................916B.1.3

Defined classes and methods..............................................................................................917B.1.4

FlDataCollection class..........................................................................................................918B.1.5

FlDataManager class............................................................................................................922B.1.6

FlDataRecord class..............................................................................................................924B.1.7

FlProperties class.................................................................................................................926B.1.8

FlPythonString class.............................................................................................................927B.1.9

Python examples..................................................................................................................929B.1.10

Reserved Words..................................................................................................................935Appendix C

About Reserved Words........................................................................................................935C.1

Data Quality Appendix........................................................................................................939Appendix D

Address Cleanse reference..................................................................................................939D.1

Country ISO codes and assignment engines........................................................................939D.1.1

2010-11-2311

Contents

Information codes (Global Address Cleanse).......................................................................959D.1.2

Status codes (Global Address Cleanse)...............................................................................962D.1.3

Quality codes (Global Address Cleanse)..............................................................................967D.1.4

Status Codes (USA Regulatory Address Cleanse)...............................................................968D.1.5

USA Regulatory Address Cleanse transform fault codes.....................................................971D.1.6

About ShowA and ShowL (USA and Canada)......................................................................973D.1.7

Data Cleanse reference.......................................................................................................982D.2

Data parsing details..............................................................................................................982D.2.1

Regular expressions.............................................................................................................988D.2.2

Creating regular expressions................................................................................................992D.2.3

Define a pattern...................................................................................................................993D.2.4

Rule Files.............................................................................................................................999D.2.5

Creating a rule file................................................................................................................999D.2.6

Rule file organization..........................................................................................................1000D.2.7

Parsing rule organization....................................................................................................1001D.2.8

User-defined pattern matching (UDPM).............................................................................1013D.2.9

Overview of UDPM............................................................................................................1014D.2.10

Working with the pattern file...............................................................................................1015D.2.11

Alternate expressions........................................................................................................1017D.2.12

Data Cleanse migration tools.............................................................................................1018D.2.13

Geocoder reference...........................................................................................................1023D.3

Index 1025

2010-11-2312

Contents

Introduction

1.1 Welcome to SAP BusinessObjects Data Services

1.1.1 Welcome

SAP BusinessObjects Data Services XI Release 3 provides data integration and data quality processesin one runtime environment, delivering enterprise performance and scalability.

The data integration processes of SAP BusinessObjects Data Services allow organizations to easilyexplore, extract, transform, and deliver any type of data anywhere across the enterprise.

The data quality processes of SAP BusinessObjects Data Services allow organizations to easilystandardize, cleanse, and consolidate data anywhere, ensuring that end users are always working withinformation that's readily available, accurate, and trusted.

1.1.2 Documentation set for SAP BusinessObjects Data Services

You should become familiar with all the pieces of documentation that relate to your SAP BusinessObjectsData Services product.

What this document providesDocument

Information about available SAP BusinessObjects Data Ser-vices books, languages, and locationsDocumentation Map

Highlights of new key features in this SAP BusinessObjectsData Services release. This document is not updated for ser-vice pack or fix pack releases.

Release Summary

Important information you need before installing and deployingthis version of SAP BusinessObjects Data ServicesRelease Notes

2010-11-2313

Introduction


An introduction to SAP BusinessObjects Data ServicesGetting Started Guide

Information about and procedures for installing SAP Busines-sObjects Data Services in a Windows environmentInstallation Guide for Windows

Information about and procedures for installing SAP Busines-sObjects Data Services in a UNIX environmentInstallation Guide for UNIX

Guidelines and options for migrating applications includinginformation on multi-user functionality and the use of the centralrepository for version control

Advanced Development Guide

Information about how to use SAP BusinessObjects DataServices DesignerDesigner Guide

Information for third-party developers to access SAP Busines-sObjects Data Services functionality using web services andAPIs

Integrator's Guide

Information about how to use SAP BusinessObjects DataServices AdministratorManagement Console: Administrator Guide

Information about how to use SAP BusinessObjects DataServices Metadata Reports

Management Console: Metadata ReportsGuide

Release-specific product behavior changes from earlier ver-sions of SAP BusinessObjects Data Services to the latest re-lease. This manual also contains information about how tomigrate from SAP BusinessObjects Data Quality Managementto SAP BusinessObjects Data Services

Migration Considerations

Information about how to improve the performance of SAPBusinessObjects Data ServicesPerformance Optimization Guide

Detailed reference material for SAP BusinessObjects DataServices DesignerReference Guide

2010-11-2314

Introduction


A compiled master PDF of core SAP BusinessObjects DataServices books containing a searchable master table of con-tents and index: Getting Started Guide Installation Guide for Windows Installation Guide for UNIX Designer Guide Reference Guide Management Console: Metadata Reports Guide Management Console: Administrator Guide Performance Optimization Guide Advanced Development Guide Supplement for J.D. Edwards Supplement for Oracle Applications Supplement for PeopleSoft Supplement for Siebel Supplement for SAP

Technical Manuals

A step-by-step introduction to using SAP BusinessObjectsData ServicesTutorial

In addition, you may need to refer to several Adapter Guides and Supplemental Guides.


Information about how to install, configure, and use the SAP BusinessObjectsData Services Salesforce.com Adapter Interface

Salesforce.com Adapter Inter-face

Information about interfaces between SAP BusinessObjects Data Servicesand J.D. Edwards World and J.D. Edwards OneWorldSupplement for J.D. Edwards

Information about the interface between SAP BusinessObjects Data Servicesand Oracle Applications

Supplement for Oracle Applica-tions

Information about interfaces between SAP BusinessObjects Data Servicesand PeopleSoftSupplement for PeopleSoft

Information about interfaces between SAP BusinessObjects Data Services,SAP Applications, and SAP NetWeaver BWSupplement for SAP

Information about the interface between SAP BusinessObjects Data Servicesand SiebelSupplement for Siebel

2010-11-2315

Introduction

1.1.3 Accessing documentation

You can access the complete documentation set for SAP BusinessObjects Data Services in severalplaces.

1.1.3.1 Accessing documentation on Windows

After you install SAP BusinessObjects Data Services, you can access the documentation from the Startmenu.

1. Choose Start > Programs > SAPBusinessObjects XI 3.2 > SAPBusinessObjects Data Services> Data Services Documentation.

Note:Only a subset of the documentation is available from the Start menu. The documentation set for thisrelease is available in LINK_DIR\Doc\Books\en.

2. Click the appropriate shortcut for the document that you want to view.

1.1.3.2 Accessing documentation on UNIX

After you install SAP BusinessObjects Data Services, you can access the online documentation bygoing to the directory where the printable PDF files were installed.

1. Go to LINK_DIR/doc/book/en/.

2. Using Adobe Reader, open the PDF file of the document that you want to view.

1.1.3.3 Accessing documentation from the Web

You can access the complete documentation set for SAP BusinessObjects Data Services from the SAPBusinessObjects Technical Customer Assurance site.

1. Go to http://help.sap.com.2. Click SAP BusinessObjects at the top of the page.3. Click All Products in the navigation pane on the left.

2010-11-2316

Introduction

You can view the PDFs online or save them to your computer.

1.1.4 SAP BusinessObjects information resources

A global network of SAP BusinessObjects technology experts provides customer support, education,and consulting to ensure maximum business intelligence benefit to your business.

Useful addresses at a glance:

ContentAddress

Information about Technical Customer Assuranceprograms, as well as links to technical articles,downloads, and online forums. Consulting servicescan provide you with information about how SAPBusinessObjects can help maximize your businessintelligence investment. Education services canprovide information about training options andmodules. From traditional classroom learning totargeted e-learning seminars, SAP BusinessOb-jects can offer a training package to suit yourlearning needs and preferred learning style.

Customer Support, Consulting, and Educationservices

http://service.sap.com/

Get online and timely information about SAPBusinessObjects Data Services, including tips andtricks, additional downloads, samples, and muchmore. All content is to and from the community, sofeel free to join in and contact us if you have asubmission.

SAP BusinessObjects Data Services Community

https://www.sdn.sap.com/irj/boc/ds

Search the SAP BusinessObjects forums on theSAP Community Network to learn from other SAPBusinessObjects Data Services users and startposting questions or share your knowledge withthe community.

Forums on SCN (SAP Community Network

https://www.sdn.sap.com/irj/scn/forums

Blueprints for you to download and modify to fityour needs. Each blueprint contains the necessarySAP BusinessObjects Data Services project, jobs,data flows, file formats, sample data, template ta-bles, and custom functions to run the data flows inyour environment with only a few modifications.

Blueprints

http://www.sdn.sap.com/irj/boc/blueprints

SAP BusinessObjects product documentation.Product documentation

http://help.sap.com/businessobjects/

2010-11-2317

Introduction

ContentAddress

Get information about supported platforms for SAPBusinessObjects Data Services.

In the left panel of the window, navigate to Docu-mentation > Supported Platforms/PARs > SAPBusinessObjects Data Services > SAP Busines-sObjects Data Services XI 3.x. Click the appropri-ate link in the main window.

Supported Platforms (formerly the Products Avail-ability Report or PAR)

https://service.sap.com/bosap-support

1.2 Overview of this guide

1.2.1 About this guide

The Data Services Reference Guide provides a detailed information about the objects, data types,transforms, and functions in the Designer.

For source-specific information, such as information pertaining to a particular back-office application,refer to the supplement for that application.

1.2.2 Who should read this guide

This and other SAP BusinessObjects Data Services software documentation assume the following:

You are a software developer, consultant, or database administrator working on data extraction,data warehousing, data integration, or data quality.

You understand your source and target data systems, DBMS, legacy systems, business intelligence,and messaging concepts.

You understand your organization's data needs.

You are familiar with SQL (Structured Query Language). If you are interested in using this software to design real-time processing, you are familiar with:

DTD and XML Schema formats for XML files

Publishing Web Services (WSDL, HTTP/S and SOAP protocols, and so on.)

2010-11-2318

Introduction

You are familiar with the installation environments: Microsoft Windows or UNIX.

2010-11-2319

Introduction

2010-11-2320

Introduction

Objects

This section provides a reference of detailed information about the objects, data types, transforms, andfunctions in the Designer.

Note:For information about source-specific objects, consult the reference chapter in the supplement documentfor that source.

Related Topics Characteristics of objects Descriptions of objects

2.1 Characteristics of objects

This section discusses common characteristics of all the objects.

Related Topics Object classes Object options, properties, and attributes

2.1.1 Object classes

An object's class determines how you create and retrieve the object. There are two classes of objects:

Reusable objects

Single-use objects

Related Topics Reusable objects Single-use objects

2010-11-2321

Objects

2.1.1.1 Reusable objects

After you define and save a reusable object, SAP BusinessObjects Data Services stores the definitionin the repository. You can then reuse the definition as often as necessary by creating calls to thedefinition.

Most objects created in the software are available for reuse. You access reusable objects through theobject library.

A reusable object has a single definition; all calls to the object refer to that definition. If you change thedefinition of the object in one place, and then save the object, the change is reflected to all other callsto the object.

A data flow, for example, is a reusable object. Multiple jobs, such as a weekly load job and a daily loadjob, can call the same data flow. If the data flow is changed, both jobs call the new version of the dataflow.

When you drag and drop an object from the object library, you are creating a new reference (or call)to the existing object definition.

You can edit reusable objects at any time independent of the current open project. For example, if youopen a new project, you can go to the object library, open a data flow, and edit it. The object will remain"dirty" (that is, your edited changes will not be saved) until you explicitly save it.

Functions are reusable objects that are not available in the object library. The software provides accessto these objects through the function wizard wherever they can be used.

Some objects in the object library are not reusable in all instances:

Datastores are in the object library because they are a method for categorizing and accessingexternal metadata.

Built-in transforms are "reusable" in that every time you drop a transform, a new instance of thetransform is created.

"Saving" a reusable object means storing the language that describes the object to the repository. Thedescription of a reusable object includes these components:

Properties of the object

Options for the object

Calls this object makes to other objects

Definition of single-use objects called by this object

If an object contains a call to another reusable object, only the call to the second object is saved, notchanges to that object's definition.

The description is stored even if the object does not validate.

2010-11-2322

Objects

Objects are saved without prompting you:

When you import an object into the repository.

When you finish editing:

Datastores

Flat file formats

XML Schema or DTD formats

You can explicitly save the reusable object currently open in the workspace by choosing Save from theProject menu. If a single-use object is open in the workspace, the Save command is not available.

To save all objects in the repository that have changes, choose Save All from the Project menu.

You are prompted to save all objects that have changes when you execute a job and when you exit theDesigner.

2.1.1.2 Single-use objects

Single-use objects appear only as components of other objects. They operate only in the context inwhich they were created.

"Saving" a single-use object means storing the language that describes the object to the repository.The description of a single-use object can only be saved as part of the reusable object that calls thesingle-use object.

The description is stored even if the object does not validate.

2.1.2 Object options, properties, and attributes

Each object is associated with a set of options, properties, and attributes:

Options control the operation of an object. For example, in a datastore, an option is the name ofthe database to which the datastore connects.

Properties document an object. For example, properties include the name, description of anobject, and the date on which it was created. Properties merely describe an object; they do not affectan object's operation.

To view properties, right-click an object and select Properties.

Attributes provide additional information about an object. Attribute values may also affect anobject's behavior.

2010-11-2323

Objects

To view attributes, double-click an object from an editor and click the Attributes tab.

2.2 Descriptions of objects

This section describes each object and tells you how to access that object.

The following table lists the names and descriptions of the objects.

DescriptionClassObject

Describes a flow, part of a flow, or a diagram in the workspace.Single-useAnnotation

Specifies the steps to execute if anerror occurs in a given excep-tion group while a job is running.Single-useCatch

Defines the format for a COBOL copybook file source.ReusableCOBOL copybook fileformat

Specifies the steps to execute based on the result of a condition.Single-useConditional

Defines activities that the software executes at a given time in-cluding error, monitor and trace messages.

Jobs can be dropped only in the project tree. The object createdis a direct reference to the object in the object library. Only onereference to a job can exist in the project tree at one time.

ReusableBatch Job

Specifies the requirements for extracting, transforming, andloading data from sources to targets.ReusableData flow

Specifies the connection information needed to access adatabase or other data source. Cannot be dropped.Single-useDatastore

Available in certain adapter datastores, documents are datastructures that can support complicated nested schemas.ReusableDocument

A description of an XML file or message. Indicates the formatan XML document reads or writes.ReusableDTD

2010-11-2324

Objects


Defines the format for an Excel workbook source.ReusableExcel workbook for-mat

Indicates how flat file data is arranged in a source or target file.ReusableFile format

Returns a value.ReusableFunction

Records information about a particular execution of a single job.Single-useLog

Available in certain adapter datastores, message functions canaccommodate XML messages when properly configured.ReusableMessage function

Available in certain adapter datastores, outbound messages areXML-based, hierarchical communications that real-time jobs canpublish to adapters.

ReusableOutbound message

Groups jobs for convenient access.Single-useProject

Retrieves a data set that satisfies conditions that you specify.Single-useQuery transform

Defines activities that the software executes on-demand.

Real-time jobs are created in the Designer, then configured andrun as services associated with an Access Server in the Admin-istrator. Real-time jobs are designed according to data flowmodel rules and run as a request-response system.

ReusableReal-time job

Evaluates expressions, calls functions, and assigns values tovariables.Single-useScript

An object from which the software reads data in a data flow.Single-useSource

2010-11-2325

Objects


Indicates an external DBMS table for which metadata has beenimported, or the target table into which data is or has beenplaced.

A table is associated with its datastore; it does not exist indepen-dently of a datastore connection. A table retrieves or stores databased on the schema of the table definition from which it wascreated.

ReusableTable

An object in which the software loads extracted and transformeddata in a data flow.Single-useTarget

A new table you want added to a database.

All datastores except SAP datastores have a default templatethat you can use to create any number of tables in the datastore.

The software creates the schema for each instance of a templatetable at runtime. The created schema is based on the dataloaded into the template table.

ReusableTemplate table

Performs operations on data sets.

Requires zero or more data sets; produces zero or one data set(which may be split).

ReusableTransform

Introduces a try/catch block.Single-useTry

Repeats a sequence of steps as long as a condition is true.Single-useWhile loop

Orders data flows and operations supporting data flows.ReusableWork flow

A batch or real-time source or target. As a source, an XML filetranslates incoming XML-formatted data into data that the soft-ware can process. As a target, an XML file translates the dataproduced by a data flow, including nested data, into an XML-formatted file.

Single-useXML file

2010-11-2326

Objects


A real-time source or target. As sources, XML messages translateincoming XML-formatted requests into data that a real-time jobcan process. As targets, XML messages translate the result ofthe real-time job, including hierarchical data, into an XML-format-ted response and sends the messages to the Access Server.

Single-useXML message

A description of an XML file or message. Indicates the formatan XML document reads or writes.ReusableXML Schema

A target that creates an XML file that matches a particular inputschema. No DTD or XML Schema is required.Single-useXML template

2.2.1 Annotation

ClassSingle-use

AccessClick the annotation icon in the tool palette, then click in the workspace.

DescriptionAnnotations describe a flow, part of a flow, or a diagram in a workspace. An annotation is associatedwith the job., work flow, or data flow where it appears. When you import or export that job, work flow,or data flow, you import or export associated annotations.

Note:An annotation has no options or properties.

Related Topics Designer Guide: Creating annotations

2010-11-2327

Objects

2.2.2 Batch Job

ClassReusable

Access In the object library, click the Jobs tab.

In the project area, select a project and right-click Batch Job.

DescriptionA batch job is a set of objects that you can schedule and execute together. To execute the steps of anyobject, the object must be part of a job.

A batch job can contain the following objects:

Data flows

Sources

Transforms

Targets

Work flows

Scripts

Conditionals

Try/catch blocks

While Loops

You can run batch jobs such that you can automatically recover from jobs that do not executesuccessfully. During automatic recovery, SAP BusinessObjects Data Services retrieves the results fromsteps that were successfully completed in the previous run and executes all other steps. Specifically,the software retrieves results from the following types of steps:

Work flows

Data flows

Script statements

Custom functions (stateless type only)

SQL function

2010-11-2328

Objects

EXEC function

get_env function

rand function

sysdate function

systime function

Batch jobs have the following built-in attributes:

DescriptionAttribute

The name of the object. This name appears on the object in the object libraryand in the calls to the object.Name

Your description of the job.Description

The date when the object was created.Date created

Batch and real-time jobs have properties that determine the information collected and logged whenrunning the job. You can set the default properties that apply each time you run the job or you can setexecution (run-time) properties that apply for a particular run. Execution properties override defaultproperties.

To set default properties, select the job in the project area or the object library, right-click, and chooseProperties to open the Properties window.

Execution properties are set as you run a job. To set execution properties, right-click the job in theproject area and choose Execute. The Designer validates the job and opens the Execution Propertieswindow.

You can set several types of execution properties:

Execution options

Trace properties

Global variables

Substitution parameters

Related Topics Designer Guide: Setting global variable values

2010-11-2329

Objects

2.2.2.1 Execution Options

Use execution options to help capture and diagnose errors using log, auditing, statistics collection, orrecovery options.

Log information goes to one of three files (in the $LINK_DIR\log\ Job Server name\repositoryname directory):

Monitor log file

Trace log file

Error log file

When you execute a job, you can also select a system configuration and a Job Server or server groupfrom the Execution Options tab of the Execution Properties window.

Select the Execution Options tab to set the following options.

DescriptionOptions

Select this check box if you want to display cache statistics in the Performance Monitorin the Administrator. (The default is cleared.)

Note:Use this option only if you want to look at the cache size.

For more information, see Monitoring and tuning caches in the Performance Optimiza-tion Guide.

For more information, see the Using Caches chapter in the Performance OptimizationGuide.

Collect statistics formonitoring

Select this check box if you want to collect statistics that the Data Services optimizerwill use to choose an optimal cache type (in-memory or pageable). This option is notselected by default.

For more information, see Caching sources in the Performance Optimization Guide.

For more information, see the Using Caches chapter in the Performance OptimizationGuide.

Collect statistics foroptimization

2010-11-2330

Objects

DescriptionOptions

Select this check box if you do not want to collect data validation statistics for anyvalidation transforms in this job. (The default is cleared.)

For more information about data validation statistics, see Data Validation dashboardsSettings control panel" in the Management Console: Metadata Reports Guide.

For more information about data validation statistics, see the Data Validation DashboardReports chapter in the Management Console: Metadata Reports Guide.

Disable data valida-tion statistics collec-tion

Select the level within a job that you want to distribute to multiple job servers for pro-cessing: Job - The whole job will execute on an available Job Server.

Data flow - Each data flow within the job can execute on an available Job Server.

Sub data flow - Each sub data flow (can be a separate transform or function) withina data flow can execute on an available Job Server.

For more information, see Using grid computing to distribute data flows executionin the Performance Optimization Guide.

For more information, see the Distributing Data Flow Execution chapter in thePerformance Optimization Guide.

Distribution level

Clear this check box if you do not want to collect audit statistics for this specific jobexecution. (The default is selected.)

For more information about auditing, see Using Auditing in the Designer Guide.

For more information about auditing, see the Data Assessment chapter in the DesignerGuide.

Enable auditing

(Batch jobs only) Select this check box to enable the automatic recovery feature. Whenenabled, the software saves the results from completed steps and allows you to resumefailed jobs. You cannot enable the automatic recovery feature when executing a jobin data scan mode.

See Automatically recovering jobs in the Designer Guide for information about therecovery options.

See the Recovery Mechanisms chapter in the Designer Guide for information aboutthe recovery options.

This property is only available as a run-time property. It is not available as a defaultproperty.

Enable recovery

2010-11-2331

Objects

DescriptionOptions

Generates and exports all specified job reports to the location specified in the ReportServer Configuration node in the Management Console Administrator. By default, thereports are exported to $LINK_DIR\DataQuality\reports\repository\job.

Export Data Qualityreports

Select the Job Server or server group to execute this job. A Job Server is defined bya host name and port while a server group is defined by its name. The list containsJob Servers and server groups linked to the job's repository.

For an introduction to server groups, see Server group architecture in the ManagementConsole: Administrator Guide.

For an introduction to server groups, see the Using Server Groups chapter in theManagement Console: Administrator Guide.

When selecting a Job Server or server group, remember that many objects in theDesigner have options set relative to the Job Server's location. For example: Directory and file names for source and target files

Bulk load directories

Job Server or Servergroup

Enter the number of rows processed before the software writes information to themonitor log file and updates job events. The software writes information about thestatus of each source, target, or transform.

For example, if you enter 1000, the logs update after processing 1,000 rows.

The default is 1000. When setting the value, you must evaluate performance improve-ments gained by making fewer calls to the operating system against your ability tofind errors quickly. With a higher monitor sample rate, more data collects before callingthe operating system to open the file: performance improves. However, with a highermonitor rate, more time passes before you are able to see any errors.

Note:If you use a virus scanner on your files, exclude the log from the virus scan. Otherwise,the virus scan analyzes the log repeated during the job execution, which causes aperformance degradation.

Monitor sample rate

(# of rows)

Select this check box to print all trace messages to the trace log file for the currentJob Server.

Selecting this option overrides the trace properties set on the Trace tab.

Print all trace mes-sages

2010-11-2332

Objects

DescriptionOptions

(Batch Job only) Select this check box to resume a failed job. The software retrievesthe results from any steps that were previously executed successfully and re-executesany other steps.

This option is a run-time property. This option is not available when a job has not yetbeen executed or when recovery mode was disabled during the previous run.

Recover from lastfailed execution

Select the system configuration to use when executing this job. A system configurationdefines a set of datastore configurations, which define the datastore connections.

For more information, see Creating and managing multiple datastore configurationsin the Designer Guide.

For more information, see the Datastores chapter in the Designer Guide.

If a system configuration is not specified, the software uses the default datastoreconfiguration for each datastore.

This option is a run-time property. This option is only available if there are systemconfigurations defined in the repository.

System configuration

Select this check box if you want the optimizer to use the cache statistics collected ona previous execution of the job. (The default is selected.)

For more information, see Monitoring and tuning caches in the Performance Optimiza-tion Guide.

For more information, see Chapter 6: Using Caches in the Performance OptimizationGuide.

Use collected statis-tics

2.2.2.2 Trace properties

Use trace properties to select the information that SAP BusinessObjects Data Services monitors andwrites to the trace log file during a job. The software writes trace messages to the trace log associatedwith the current Job Server and writes error messages to the error log associated with the current JobServer.

To set trace properties, click the Trace tab. To turn a trace on, select the trace, click Yes in the Valuelist, and click OK. To turn a trace off, select the trace, click No in the Value list, and click OK.

You can turn several traces on and off.

2010-11-2333

Objects

DescriptionTrace

Writes messages exchanged between the Access Server and a service provider,including: The registration message, which tells the Access Server that the service provider

is ready

The request the Access Server sends to the service to execute

The response from the service to the Access Server

Any request from the Access Server to shut down

Access Server Commu-nication

Writes messages for Substitution Parameter and SDK Transforms: Substitution parameters - Writes trace messages such as substitution configuration

used and the actual value substituted for each substitution parameter. SDK transforms - Writes transform-specific information specified in the form of

XML. This information can be hierarchical. At runtime this XML information isextracted or assembled. The trace messages write the assembled XML.

Assemblers

Writes a message when auditing: Collects a statistic at an audit point

Determines if an audit rule passes or failsAudit Data

Writes a message when the data flow starts, when the data flow successfully finishes,or when the data flow terminates due to error.

This trace also reports when the bulk loader starts, any bulk loader warnings occur,and when the bulk loader successfully completes.

Data Flow

Writes a message when reading IDoc files including: start reading the IDoc file stop reading the IDoc file result of the IDoc file validation

IDoc file reader

Writes a message for every row retrieved from the memory table.Memory Source

Writes a message for every row inserted into the memory table.Memory Target

For consulting and customer assurance use.Optimized Dataflow

2010-11-2334

Objects

DescriptionTrace

Writes a message related to RFC calls including: start of RFC call end of RFC call a message for each record received from the software for the RFC call

RFC Function

Writes a message when a transform imports or exports a row.Row

Writes messages from readers of SAP system tables including: start reading from table stop reading from table start of connection to SAP system where table is present end of connection to SAP system

SAP Table Reader

Writes a message when the software runs a script or invokes a script function.Specifically, this trace links a message when: The script is called. Scripts can be started any level from the job level down to

the data flow level. Additional (and separate) notation is made when a script iscalled from within another script.

A function is called by the script.

The script successfully completes.

Scripts and ScriptFunctions

Writes a message when the job description is read from the repository, when the jobis optimized, and when the job runs.Session

Writes data retrieved before SQL functions: Every row retrieved by the named query before the SQL is submitted in the

key_generation function

Every row retrieved by the named query before the SQL is submitted in the lookupfunction (but only if PRE_LOAD_CACHE is not specified).

When mail is sent using the mail_to function.

SQL Functions

2010-11-2335

Objects

DescriptionTrace

Writes a message when the bulk loader: Starts

Submits a warning message

Completes successfully

Completes unsuccessfully, if the Clean up bulk loader directory after loadoption is selected

Additionally, for Microsoft SQL Server and Sybase ASE, writes when the SQ

Xi321 Ds Reference En

Documents

Transcript of Xi321 Ds Reference En