Informatica PowerMart / PowerCenter 6.x

Upgrade Features

Ted Williams

Informatica Corporation, 2002. All rights reserved.

2

Informatica PowerCenter 6 Architecture

Not Shown: Client ODBC Connections for Source and Target metadata

HeterogeneousTargets

HeterogeneousSources

Repository

Targets

RepositoryServer

RepositoryAgent

Sources

Repository Designer Workflow Workflow Manager Monitor Manager

3

Repository Server

Each repository has an independent architecture for the management of the physical repository tables

Components: one Repository Server and a Repository Agent for each repository

Repository

RepositoryRepositoryServerServer

RepositoryRepositoryAgent(s)Agent(s) Repository Repository

Manager Manager

ServerServer

Client overhead for repository management is greatly reduced by the Repository Server

4

Repository Server

The Repository Server manages connections to the repository from client applications

The Repository Server can manage multiple repositories on different machines on a network

For each repository it manages, the Repository Server uses one process, called the Repository Agent, to insert, update, and fetch objects from the repository database tables

The Repository Server maintains object consistency by controlling object locking

The Repository Server runs on the same system running the Repository Agent

5

Repository Server Console

6

Designer Enhancements

Heterogeneous Targets ReplaceChr and ReplaceStr Copy and Paste (Designer) Sorter Object Transaction Control Object Cascading code changes More robust migration via XML import/export (Replace option) Compare Objects Compare Mappings Highlight Path Links Removed restrictions on Mapplets Dynamic Lookup Cache Update support Expand Mapplets for visualization in Mappings True Flat-file target objects Metadata Extensions for better reporting

7

Session Manager to Workflow Manager Change

• Represents the largest change to date

• Similar in structure to Designer

• New objects/tasks

• Concept of Links

• Better monitoring capabilities

8

Workflow ManagerTasksTasks

LinksLinks

Variables/Variables/EventsEvents

Work Work FoldersFolders

9

New Connection Features

Connection Dialogs

Replace Connections

Register Server

Assign to Workflows

10

Connection DialogsType FilterType Filter

Copy AsCopy As

Environmental Environmental OptionsOptions

- Type filters allow for easy - Type filters allow for easy viewing of Database specific viewing of Database specific connectionsconnections

- Copy As feature allow for - Copy As feature allow for quick propagation of main quick propagation of main connection info (maybe only connection info (maybe only user and password is user and password is different)different)

- Most significant change is - Most significant change is the ability to assign a the ability to assign a specific rollback segment specific rollback segment and/or prepare database and/or prepare database with environment SQLwith environment SQL

11

Replace Connections

Replace Replace ConnectionsConnections

With folder closed, click on the Replace Connections button.

This brings up the sub-dialog that allows a From-To operation to allow the global replace of one connection with another in the sessions.

12

Register Server

Similar to Relational Connection dialog

Same parameters apply with the exception of the new variables for Workflow logs.

13

Assign to Workflows

While folders are closed it is possible to assign choose sessions to a particular server

This dialog allows for individual or globally select sessions to be assigned to run on a particular server

14

New Objects (Tasks)

Local Tasks – Sessions

Commands

Email

Decision

Assignment

Timer

Control

Event Raise

Event Wait

Global Tasks – Sessions

Commands

Email

Tasks are now the default units of work for building the workflow. Global tasks are reusable across workflows. Local tasks are independent and self-contained within workflows.

15

Sessions (Updates)

GeneralGeneral

First page allows for options to treat conditional links attached to the object as AND/OR functionality. Also control option to fail the parent (container) if task fails or does not run. Disabling a task in a workflow allows the task to be skipped instead of having to remove it.

Updated Updated parametersparameters

16

Sessions (Updates)

PropertiesProperties

Addition to this page is the option to choose a different connection to resolve $Source and $Target used in lookups.

**Limitation - You can use $DBConnectionVariable but it will only resolve if the parameter is evaluated with the appropriate Source or Target Connection

If used when sourcing and targeting flat-files the parameter will not resolve.

Updated Updated connection connection resolutionresolution

17

Sessions (Updates)

Config ObjectConfig Object

This area is where typical allocation of memory occurs, log file options and error handling This area is where typical allocation of memory occurs, log file options and error handling strategies. The upgrade here is with the concept of maintaining common setting in a config strategies. The upgrade here is with the concept of maintaining common setting in a config object which is setup separately and chosen from a drop-down list.object which is setup separately and chosen from a drop-down list.

Drop-down for Drop-down for common common settingssettings

18

Config Object

Config ObjectConfig Object

By creating a session configuration object you can cut down on repetitively assigning memory By creating a session configuration object you can cut down on repetitively assigning memory and log options. When migrating sessions you can optionally migrate configurations or maintain and log options. When migrating sessions you can optionally migrate configurations or maintain environment specific settings to be reassigned.environment specific settings to be reassigned.

Setup for reusable Setup for reusable Configuration objectConfiguration object

19

Sessions (Updates)

SourcesSources

This area has primarily just been subdivided into sub tabs. The main addition is the filter drop-This area has primarily just been subdivided into sub tabs. The main addition is the filter drop-down that allows for either viewing all source object settings or by individual source object. down that allows for either viewing all source object settings or by individual source object. Instead of ‘filelist’ option it is either direct or indirect (filelist)Instead of ‘filelist’ option it is either direct or indirect (filelist)

FilterFilter

20

Sessions (Updates)

TargetsTargets

Same subdivisions as sources. However in the Properties sub tab is where the ‘Target Options’ Same subdivisions as sources. However in the Properties sub tab is where the ‘Target Options’ are defined. The writers sub tab preserves the legacy operation of writing to a flat-file while are defined. The writers sub tab preserves the legacy operation of writing to a flat-file while using a database specific target (as opposed to the new flat-file target). using a database specific target (as opposed to the new flat-file target).

FilterFilter

21

Source Target Connections

ConnectionsConnections

The drop-down list allows to select any available connection as well as specifically define the use The drop-down list allows to select any available connection as well as specifically define the use of a Connection Variable of a Connection Variable

Connection Connection dialogdialog

22

Sessions (Updates)

ComponentsComponents

The area where commands or email unique to this object can be defined. You can alternately The area where commands or email unique to this object can be defined. You can alternately select a reusable task to use as well. select a reusable task to use as well.

Choice of Choice of reusable or reusable or

local commandlocal command

23

Non-Reusable Commands

ComponentsComponents

Regardless of reusable or non-reusable it is necessary to name the object since there is Regardless of reusable or non-reusable it is necessary to name the object since there is potential to promote it.potential to promote it.

Option for local Option for local or reusableor reusable

Name of Name of command command

objectobject

24

Non-Reusable Commands

ComponentsComponents

The properties tab allows for error control for commands/tasksThe properties tab allows for error control for commands/tasks

Error Control for Error Control for multiple multiple

commands/taskscommands/tasks

25

Non-Reusable Commands

ComponentsComponents

The Command tab allows for editing of commands (obviously)The Command tab allows for editing of commands (obviously)

Command editorCommand editor

26

Reusable Commands

ComponentsComponents

A non-reusable command can be promoted to reusable or else you can select one from available A non-reusable command can be promoted to reusable or else you can select one from available tasks.tasks.

Reusable Reusable Command Command

Selection DialogSelection Dialog

27

Reusable Commands

ComponentsComponents

Limitation – a Limitation – a non-reusablenon-reusable and and reusablereusable command command cancan resolve Server Variables but only from resolve Server Variables but only from within the Components tab of the session. A command can within the Components tab of the session. A command can NOTNOT resolve Server Variables if resolve Server Variables if used in an external used in an external Command TaskCommand Task. Understand before promoting.. Understand before promoting.

LimitationLimitation

28

Sessions (Updates)

TransformationsTransformations

The transformations area preserves all the previous capabilities. The update is the addition of The transformations area preserves all the previous capabilities. The update is the addition of the main instance filter and another filter button that only displays transformation where overrides the main instance filter and another filter button that only displays transformation where overrides can be performed. Also this is where you can input the NEW pre/post SQL commands.can be performed. Also this is where you can input the NEW pre/post SQL commands.

Filter for modifiable Filter for modifiable transformationstransformations

Instance FilterInstance Filter

29

Sessions (Updates)

PartitionsPartitions

New partitioning scheme allows for repartitioning after Source Qualifier at almost any other New partitioning scheme allows for repartitioning after Source Qualifier at almost any other transformation object in the mapping. There are four main partition types Pass Through, Round transformation object in the mapping. There are four main partition types Pass Through, Round Robin, Hash Auto Keys, Hash User Keys.Robin, Hash Auto Keys, Hash User Keys.

Add Partition pointsAdd Partition points

Change Partition TypeChange Partition Type

30

Sessions (Updates)

Metadata ExtensionsMetadata Extensions

Metadata extensions are available to individual object/tasks or can be made reusable and global. Metadata extensions are available to individual object/tasks or can be made reusable and global. These columns can not be auto-populated with functions but they do allow the extendibility for These columns can not be auto-populated with functions but they do allow the extendibility for more reporting options (e.g., Developer Name, Version, Creation Date, etc.)more reporting options (e.g., Developer Name, Version, Creation Date, etc.)

Metadata columnMetadata column

Promote to Promote to reusablereusable

31

Command Tasks

CommandCommand

The command object can be created globally under the Task Developer. It can also be The command object can be created globally under the Task Developer. It can also be promoted here from within a mapping. promoted here from within a mapping.

Created in Task Created in Task DeveloperDeveloper

32

Command Tasks

CommandCommand

The properties section homes the ability to either run all commands regardless or run them if The properties section homes the ability to either run all commands regardless or run them if each previous command completes. Commands tab is where the actual commands are created. each previous command completes. Commands tab is where the actual commands are created. One command per line. One command per line.

Process Control Process Control for multiple for multiple commandscommands

33

Email Tasks

EmailEmail

Email task is very similar to the command task since it can be either created in the Task Email task is very similar to the command task since it can be either created in the Task Developer or promoted from a mapping. The properties tab allows for an expression editor for Developer or promoted from a mapping. The properties tab allows for an expression editor for text creation utilizing the built-in variables.text creation utilizing the built-in variables.

Email text Email text creation dialogcreation dialog

Built-in Built-in VariablesVariables

34

Links and Conditions

DefiinitionDefiinition

Links and their underlying conditions are what provide process control to the workflow. When an Links and their underlying conditions are what provide process control to the workflow. When an attached link condition resolves to TRUE then the attached object may begin processing. There attached link condition resolves to TRUE then the attached object may begin processing. There can be no looping and links can only execute once per workflow. However more complex can be no looping and links can only execute once per workflow. However more complex branching and decisions can be made by combining multiple links to a single object or branching branching and decisions can be made by combining multiple links to a single object or branching into decision type paths. Each link has its own expression editor and can utilize upstream into decision type paths. Each link has its own expression editor and can utilize upstream resolved object variables or user-defined variables for its own evaluation.resolved object variables or user-defined variables for its own evaluation.

Link conditionLink condition

35

Links and Conditions

Object VariablesObject Variables

The default set of object variables from a session can provide more information than just a status The default set of object variables from a session can provide more information than just a status of ‘Completed’. More complex evaluation can be done for ErrorCode, StartTime, of ‘Completed’. More complex evaluation can be done for ErrorCode, StartTime, SrcSuccessRows, etc. In addition to the default object variables, User Defined variables can be SrcSuccessRows, etc. In addition to the default object variables, User Defined variables can be created and populated via parameter files or changed in the workflow via Assignment tasks. created and populated via parameter files or changed in the workflow via Assignment tasks. Also any upstream task that has completed can have it’s variables utilized in downstream link Also any upstream task that has completed can have it’s variables utilized in downstream link conditions.conditions.

Object Object VariablesVariables

36

Workflow Variables

Pre-defined VariablesPre-defined Variables

This is the list of all pre-defined task level variables available to evaluate uponThis is the list of all pre-defined task level variables available to evaluate upon

Variable Task Type Datatype ** Supported Status Returns

ABORTED

DISABLED

FAILED

NOTSTARTED

STARTED

STOPPED

SUCCEEDED

Condition Decision Task Integer

EndTime All tasks Date/time

ErrorCode All tasks Integer

ErrorMsg All tasks Nstring*

FirstErrorCode Session task Integer

FirstErrorMsg Session task Nstring*

PrevTaskStatus All tasks Integer

SrcFailedRows Session task Integer

SrcSuccessRows Session task Integer

StartTime All tasks Date/time

Status** All tasks Integer

TgtFailedRows Session tasks Integer

TgtSuccessRows Sessions Integer

TotalTransErrors Sessions Integer

* Variables of type Nstring can have a maximum length of 600 characters.

37

Workflow Variables

User-defined VariablesUser-defined Variables

Variables are created at the container level much like the mappings. (Workflows=Mappings, Variables are created at the container level much like the mappings. (Workflows=Mappings, Worklets=Mapplets). Once created values can be passed to objects within the same container Worklets=Mapplets). Once created values can be passed to objects within the same container for evaluation. (Assignment Task can modify/calculate variables)for evaluation. (Assignment Task can modify/calculate variables)

Edit VariablesEdit Variables

38

Workflow Variables

User-defined VariablesUser-defined Variables

A user-defined variable can assist in more complex evaluations. In the above example, an A user-defined variable can assist in more complex evaluations. In the above example, an external parameter file contains the number of expected rows. This in turn is evaluated against external parameter file contains the number of expected rows. This in turn is evaluated against the actual rows successfully read from an upstream session. $ signifies and is reserved for pre-the actual rows successfully read from an upstream session. $ signifies and is reserved for pre-defined variables. User defined variables should maintain $$ naming.defined variables. User defined variables should maintain $$ naming.

User Defined User Defined VariablesVariables

Pre-Defined Pre-Defined VariableVariable

39

Parameter File

FormatFormat

The format is slightly different going forward for declaring variables but legacy naming is The format is slightly different going forward for declaring variables but legacy naming is preserved with the foldername.sessionpreserved with the foldername.session

•Workflow variables: [folder name.WF:workflow name]

•Worklet variables: [folder name.WF:workflow name.WT:worklet name]

•Worklet variables in nested worklets: [folder name.WF:workflow name.WT:worklet name.WT:worklet name...]

•Session parameters, plus mapping parameters and variables:[folder name.WF:workflow name.ST:session name]

or [folder name.session name]

or [session name]

40

Assignment Task

UsageUsage

The assignment task allows for workflow variables to be manipulated, assigned or calculated and The assignment task allows for workflow variables to be manipulated, assigned or calculated and stored (either for the session run or as persistent) for downstream evaluation and condition stored (either for the session run or as persistent) for downstream evaluation and condition resolution. The Expression Section allows for the selection of the appropriate variable to resolution. The Expression Section allows for the selection of the appropriate variable to manipulate along with the assignment through an Expression Editor. In order to use it is manipulate along with the assignment through an Expression Editor. In order to use it is necessary to first create a variable through the Edit Variables dialognecessary to first create a variable through the Edit Variables dialog

Edit VariablesEdit Variables

41

Event Task

UsageUsage

Events are similar to variables. They are repository tags that are a little more flexible (cleaner) Events are similar to variables. They are repository tags that are a little more flexible (cleaner) than dealing with indicator files (although event watch can be used with indicator files). Before than dealing with indicator files (although event watch can be used with indicator files). Before utilizing the new functionality it is necessary to create these Event tags within the container that utilizing the new functionality it is necessary to create these Event tags within the container that they will be used. Events can not be monitored across workflows or worklets (even if a worklet is they will be used. Events can not be monitored across workflows or worklets (even if a worklet is part of the workflow).part of the workflow).

Edit EventsEdit Events

42

Event Task

UsageUsage

If using Event tags then an Event Raise is used in conjunction with an Event Wait. In the above If using Event tags then an Event Raise is used in conjunction with an Event Wait. In the above example two branches are executed in parallel. The second session of the lower branch will example two branches are executed in parallel. The second session of the lower branch will remain in stasis until the upper branch completes triggering the event. The lower branches remain in stasis until the upper branch completes triggering the event. The lower branches event wait task recognizes the event and allows for the second session to start.event wait task recognizes the event and allows for the second session to start.

Event RaiseEvent Raise

Event WaitEvent Wait

43

Event Raise

UsageUsage

To configure the Event Raise task the drop-down box allows for selection of the appropriate To configure the Event Raise task the drop-down box allows for selection of the appropriate user-defined Event tag. This will create an entry in the repository for a matching event wait to user-defined Event tag. This will create an entry in the repository for a matching event wait to look for.look for.

44

Event Wait

UsageUsage

The event wait allows for configuration for an Event Raise (user-defined event) or existence The event wait allows for configuration for an Event Raise (user-defined event) or existence check for an indicator file.check for an indicator file.

User Defined User Defined EventEvent

Indicator FileIndicator File

45

Event Wait

UsageUsage

The properties section of the Event Wait task allows for further definition of behavior. If your The properties section of the Event Wait task allows for further definition of behavior. If your workflow has failed/suspended after Event Raise but before the Event Wait has resolved, then workflow has failed/suspended after Event Raise but before the Event Wait has resolved, then the Enable Past Events is able to recognize that the Event has happened already. If working the Enable Past Events is able to recognize that the Event has happened already. If working with indicator files you have to ability to either delete the file or allow it to stay in case some with indicator files you have to ability to either delete the file or allow it to stay in case some downstream Event Waits are also keying on that file.downstream Event Waits are also keying on that file.

Resume/Restart Resume/Restart SupportSupport

Flat-file CleanupFlat-file Cleanup

46

Decision Task

UsageUsage

The decision task allows for True/False based branching of process ordering. The Decision task The decision task allows for True/False based branching of process ordering. The Decision task can home multiple conditions and therefore downstream links can be evaluated simply upon the can home multiple conditions and therefore downstream links can be evaluated simply upon the Decision being True or False.Decision being True or False.

**Note it is possible to have the decision based on SUCCEEDED or FAILED of previous task, **Note it is possible to have the decision based on SUCCEEDED or FAILED of previous task, however if workflow is set to suspend on error than that branch is suspended and the decision however if workflow is set to suspend on error than that branch is suspended and the decision won’t trigger on a FAILED conditionwon’t trigger on a FAILED condition

47

Control Task

UsageUsage

The control task is utilized in a branching manner to present a level of stoppage during the The control task is utilized in a branching manner to present a level of stoppage during the workflow. Consider if too many sessions have too many failed rows. The options allow for workflow. Consider if too many sessions have too many failed rows. The options allow for different levels such as failing at the object level to Aborting the whole workflow.different levels such as failing at the object level to Aborting the whole workflow.

48

Timer Task

UsageUsage

The timer task has two main ways to be utilized. The first way is by absolute time that is time The timer task has two main ways to be utilized. The first way is by absolute time that is time evaluated by server time or a user-defined variable (that contains the date/time stamp to start).evaluated by server time or a user-defined variable (that contains the date/time stamp to start).

49

Timer Task

UsageUsage

The second usage is by Relative time that offers options of time calculated from when the The second usage is by Relative time that offers options of time calculated from when the process reached this (Timer) task, from the start of the container this task, or from the start of the process reached this (Timer) task, from the start of the container this task, or from the start of the absolute top-level workflow.absolute top-level workflow.

50

Practical

Business CaseBusiness Case

Need for three sessions to wait for Need for three sessions to wait for indicator file(s) to start each one. indicator file(s) to start each one. Window of opportunity is only between Window of opportunity is only between 10AM and 2AM (next morning). If any 10AM and 2AM (next morning). If any one runs an extra session must run. At one runs an extra session must run. At cutoff time there is need to stop the cutoff time there is need to stop the process (polling - not existing runs) so process (polling - not existing runs) so that new activity does not continue that new activity does not continue between 2AM and 10AM. Workflow is between 2AM and 10AM. Workflow is scheduled to run everyday at 10AMscheduled to run everyday at 10AM

Objects Used: •Assignment Task – Assigns the appropriate cutoff time for logic•File Wait Tasks – Polls for the appropriate Indicator files•Timer Task – Assigned to start based on the variable assigned by the Assignment task•Command Tasks – After cutoff time the commands will put an indicator file to release the polling

Link Logic – The remainder of the logic is contained within the links themselves. The main sessions evaluate end time of file wait tasks to the cutoff time. If within cutoff then sessions will run. If over cutoff sessions will not run. The cutoff branch also evaluates to see if file wait tasks are running over. If they are still running then the command tasks will fire.

51

Practical_Descriptive

52

Workflow Monitor

53

Workflow Monitor Task View