Level 8 Stairway to Integration Services

8/13/2019 Level 8 Stairway to Integration Services

1/24

Advanced SSIS Workflow Management Level 8 of theStairway to Integration Services

By Andy Leonard, 2012/09/12

The Series

This article is part of the Stairway Series: Stairway to Integration Services

Integration Services is one of the most popular subsystems in SQL Server. In allows you to Extract,Transform, and Load (ETL) data between a variety of data sources and programmatically change datain any manner you can think of and script in C#.

This is the eighth article in a series entitled Stairway to Integration Services. Previous articles in theseries include:

What is SSIS? Level 1 of the Stairway to Integration Services

The SSIS Data Pump - Level 2 of the Stairway to Integration Services Adding Rows in Incremental Loads Level 3 of the Stairway to Integration Services

Updating Rows in Incremental Loads Level 4 of the Stairway to Integration Services Deleting Rows in Incremental Loads Level 5 of the Stairway to Integration Services Basic SSIS Workflow Management Level 6 of the Stairway to Integration Services Intermediate SSIS Workflow Management Level 7 of the Stairway to Integration Services

Introduction

In the previous two installments, we built a new SSIS package, took a first look at scripting andprecedence constraints in SQL Server Integration Services, and examined theMaxConcurrentExecutables package property. We examined, demonstrated, and tested On Success,On Completion, and On Failure functionality of precedence constraints.

In this article, we go deep with SSIS workflow management learning about SSIS variables and thecomplexities of precedence constraints with Expressions.

About Variables

Open the Precedence.dtsx package if its not already open. Click the SSIS dropdown menu at the topof the BIDS environment and select Variables as shown in Figure 1:


2/24

Figure 1

The Variables window displays as shown in Figure 2:

Figure 2

At the top of the Variables window, we find a toolbar for working with SSIS variables. Thesebuttons are:

The Add Variable button -

The Delete Variable button -

It is disabled at this time because we have no SSIS variables configured for the Precedence.dtsxpackage.

The third button -

is the Show System Variables button and displays System variables (variables in the Systemnamespace). Many System variables are properties of the SSIS package. Other System variables areproperties of components and objects that make up the SSIS package.

The fourth button -is the Show All Variables button. Clicking this button displays all variables in the SSIS package,regardless of scope.

The fifth button -

is the Choose Variable Columns button. As shown in Figure 3, click this button to set the visibility ofthe columns visible in the Variables window grid:


3/24

Figure 3

Variable Name, Scope, Data Type, and Value are displayed by default. You cannot deselect the

Variable Name column, but you can decide to view Scope, Data Type, and Value. You can also viewNamespace and Raise event when variable value changes.

Variables and Namespaces

There are two default namespaces in an SSIS package: System and User. You cannot add to theSystem variables, but you can add to the User variables and you can create new namespaces.Variable names must be unique with respect to Scope and Namespace. This means you can have avariable named MyVariable in the User namespace, which, when fully qualified is namedUser::MyVariable (::). In this case, User::MyVariable is scoped to thePrecedence.dtsx package. We could have another variable named MyVariable in the User2namespace, User2::MyVariable, also scoped to the Precedence.dtsx package. To create the User2namespace you need to display the namespace column and edit the text from User to User2. Whenyou access variables at the same scope having the same name but having different namespaces, you

need to fully-qualify the variable name or you will receive an error similar to:

The variable name is ambiguous because multiple variables with this name exist in differentnamespaces. Specify namespace-qualified name to prevent ambiguity.

Add a Variable

To demonstrate expressions and SSIS precedence constraints, lets add an SSIS variable to thePrecedence.dtsx package. Click the Add Variable button on the Variables window. A new Int32 data-type variable named Variable is created as shown in Figure 4:

Figure 4

Rename this variable MyBool and change its data type to Boolean, as shown in Figure 5:


4/24

Figure 5

Now we can use this variable in a precedence constraint expression.

Expressions and Precedence Constraints

Right-click the precedence constraint between Script Task 1 and Script Task 2, and click Edit toopen the Precedence Constraint Editor shown in Figure 6:

Figure 6

There are two groupbox controls on the Precedence Constraint Editor. They are labeled Constraintoptions and Multiple constraints. When I build SSIS packages, I physically orient Control Flow tasksto execute in the order top-to-bottom. One positive side-effect of positioning tasks so execution flowsfrom top-to-bottom: The arrangement of the groupboxes on the Precedence Constraint Editor alignsomewhat with the physical layout of the Precedence Constraint on the Control Flow. The Constraintoptions groupbox sets Precedence Constraint properties related to the preceding task or startpoint


5/24

of the precedence constraint. The Multiple constraints groupbox sets properties for the following taskor Precedence Constraint endpoint as shown in Figure 7:

Figure 7

On the Precedence Constraint Editor, click the Evaluation operation dropdown and examine theoptions listed in Figure 8:

Figure 8

The default option is Constraint, which is the option we used in the preceding articles in this series.Selecting Constraint allows us to configure when (or if) the next task executes based solely uponthe execution result of the previous task. The Value dropdown contains options for Constraintevaluation: Success, Failure, and Completion as shown in Figure 9:

Figure 9

Change the Evaluation operation dropdown to Expression and enter @MyBool in the Expressiontextbox as shown in Figure 10:

Figure 10

Click the Test button to validate the expression in the Expression textbox. The results should appearas shown in Figure 11:


6/24

Figure 11

Remember, the value of the SSIS Variable named MyBool is a Boolean data type and is set to adefault value of False. The expression entered in the Expression textbox must evaluate to True orFalse (a Boolean value). Because the SSIS variable MyBool is of the Boolean data type, it is perfectlysuited for the task at hand. I could edit the Expression to read @MyBool == True. The expressions@MyBool and @MyBool == True are said to be logically equivalentbecause they produce thesame results.

Once configured, close the Precedence Constraint Editor. Your Control Flow should appear as shownin Figure 12:

Figure 12

Execute the Precedence.dtsx package in the BIDS debugger by pressing the F5 key or clicking theVCR-style (or iPod-style) Play button. You should be prompted to succeed or fail Script Task 1 asshown in Figure 13:

Figure 13

It no longer matters which option you select because the precedence constraint evaluation is basedsolely on the value of the SSIS Boolean Variable named @MyBool. @MyBool is False by default, soScript Task 2 will never execute because the precedence constraint will never evaluate to True.


7/24

To create a more useful test, open the Script Task 1 Editor and click the ellipsis in theReadWriteVariables property. This displays the Select Variables window. Select the User::MyBoolvariable as shown in Figure 14:

Figure 14

Click the OK button to close the Select Variables window. Your Script Task Editor should now includethe SSIS Variable User::MyBool in the ReadWriteVariables property as shown in Figure 15:

Figure 15

Click the Edit Script button to display the Script Editor window. Edit the code in the Public Sub Mainsubroutine to as shown in Listing 1 (changes highlighted):

PublicSubMain()

DimsTaskName AsString=

Dts.Variables("TaskName").Value.ToString

DimiResponse AsInteger=_

MsgBox("Set MyBool to True?",MsgBoxStyle.YesNo,sTaskName)IfiResponse =MsgBoxResult.Yes Then

Dts.Variables("User::MyBool").Value =True

Else

Dts.Variables("User::MyBool").Value =False

EndIf

Dts.TaskResult =ScriptResults.Success

EndSub

Listing 1

Close the Script Editor and click the OK button to close the Script Task Editor. Execute thePrecedence.dtsx package in the BIDS debugger. You will now be prompted as shown in Figure 16:


8/24

Figure 16

If you respond by clicking the Yes button, you should see a message box indicating Script Task 2 hasexecuted as shown in Figure 17:

Figure 17

If you respond to the Script Task 1 message box by clicking the No button, execution should completewithout executing Script Task 2 as shown in Figure 18:

Figure 18

Lets build out this demonstration a bit. Right-click Script Task 2 and then click Copy. Then right-click inthe empty space of the Control Flow and click Paste. Script Task 2 1 should appear as shown inFigure 19:


9/24

Figure 19

Rename Script Task 2 1 to Script Task 3. Connect a new precedence constraint from Script Task 1as shown in Figure 20:

Figure 20

If you execute the package in the BIDS debugger right now, Script Task 1 will succeed regardless ofthe way you answer the prompt about the MyBool variable. When it succeeds, the precedenceconstraint connecting Script Task 1 and Script Task 3 will evaluate True, and the message boxdeclaring Script Task 3 completed will display as shown in Figure 21:

Figure 21

Lets edit the precedence constraint between Script Task 1 and Script Task 3. Double-click theprecedence constraint to open the Precedence Constraint Editor and change the Evaluation Operationto Expression. In the Expression textbox, enter !@MyBool (without the double-quotes) as shown inFigure 22:


10/24

Figure 22

The expression !@MyBool can be read Not MyBool and evaluates to True if the value of the SSISVariable User::MyBool is False. Click the OK button to close the Precedence Constraint Editor. YourControl Flow should appear similar to that shown in Figure 23:

Figure 23

I dont know about you, but to me this looks messy. The function icons are in the way some. I canadjust where they appear by moving the precedence constraints. When I do that my Control Flowappears as shown in Figure 24:

Figure 24

I like this better.

Execute the package in the BIDS debugger. If you click the Yes button when Script Task 1 prompts,

Script Task 2 will execute as shown in Figure 25:


11/24

Figure 25

If you click the No button when Script Task 1 prompts, Script Task 3 will execute as shown in Figure26:

Figure 26

Precedence constraints provide great visual feedback. Blue indicates a constraint evaluation operationset to Complete, for example. But our evaluation operation ignores the constraint altogether whatgives? Lets think about this for a minute. Constraint evaluations are Success, Failure, andCompletion. Success means the preceding task succeeded; failure means the preceding task failed.Both Success and Failure imply Completion, do they not? They do. Completion is a state that indicatesthe preceding task is done executing whether it succeeded or failed. Completion ignoresthe result ofthe preceding tasks execution; whether it was success or failure.

When I set the Precedence Constraint Evaluation Operation to Expression I am also ignoring the resultof the preceding tasks execution. I am only concerned with whether the task completed. I find the Blueline here appropriate and a good indication of the logical operation.

Copying and Disabling to Preserve Existing Work

Before we move forward, lets preserve the work weve already done. From the toolbox, drag aSequence Container onto the Control Flow as shown inFigure 27:


12/24

Figure 27

If you click in the empty space of the Control Flow and draw a box around the three script tasks, youcan drag them as a group into the Sequence Container as shown in Figure 28:

Figure 28

When you do this, the sequence container will automatically resize to surround the items dragged intoit most of the time. If it doesnt, manually resize it so it wont be visually confusing.

Right-click the Sequence Container and click Copy. Right-click in the empty space of the Control Flowand click Paste. Your Control Flow should appear as shown in Figure 29:


13/24

Figure 29

Now, right-click Sequence Container 1 the original one and click Disable. The sequence containerand its contents should appear disabled (grayed-out) as shown in Figure 30:

Figure 30

Now we can preserve the work already did while moving forward with new work. Cool.

Multiple Constraints

In Sequence Container 1, delete Script Task 1. This will also delete the precedence constraintbetween Script Task 1 and Script Task 2. Copy Script Task 2 and paste it into Sequence Container 1,

then rename the new script task Script Task 4. Move Script Task 2 so that it is parallel to Script Task4, and then connect a precedence constraint between Script Task 2 and Script Task 3. Connect a newprecedence constraint between Script Task 4 and Script Task 3 as shown in Figure 31:


14/24

Figure 31

Now, let me ask you: What has to happen in order for Script Task 3 to execute?

1. Nothing. Script Task 3 will execute when the other tasks execute.2. Script Task 3 will execute after either Script Task 4 or Script Task 2 executes and succeeds.3. Script Task 3 will execute after either Script Task 4 and Script Task 2 executes and succeeds.4. Script Task 3 will never execute.

The answer can be found by examining either of the precedence constraint editors, focusing on theMultiple Constraints groupbox as shown in Figure 32:

Figure 32

Remember, the Multiple Constraints groupbox defines how precedence constraints work at theEndPoint (the end of the precedence constraint with the arrow). When there is only one precedenceconstraint landing on a task at the endpoint, these options are moot (they behave the same way). Butwhen there is more than one constraint landing at an endpoint task as here these options are vital.

The Logical AND option is selected by default. This means all precedence constraints landing at thisendpoint task must evaluate before this task will execute. In this case, it means Script Task 4 and

Script Task 2 must complete and succeed before Script Task 3 will fire. Lets execute the package inthe BIDS debugger to validate as shown in Figure 33:


15/24

Figure 33

Script Task 3 could not start executing until afterScript Task 4 andScript Task 2 completed executionwith success. (The correct answer above was 3.) This is how Logical And Multiple Constraints works.Lets test Logical Or next. Stop the BIDS debugger and double-click either enabled precedenceconstraint to open the editor. Change the Multiple Constraints setting from Logical AND to Logical ORas shown in Figure 34:

Figure 34

Click the OK button to close the Precedence Constraint Editor. You will probably notice a change inthe appearance of both precedence constraints connected to Script Task 3. The precedenceconstraints are now both denoted by dashed lines as shown in Figure 35:


16/24

Figure 35

Because multiple constraint configurations deal with endpoint tasks, changes made to one are appliedto all precedence constraints connected to the endpoint task.

Before, when we had multiple constraints configured to Logical AND, both preceding precedenceconstraints had to evaluate before the task that followed executed. We are now configured for LogicalOR. What will need to happen before Script Task 3 will execute?


Lets test it and see! Execute the package in the BIDS debugger. At some point, your Control Flowshould appear similar to that shown in Figure 36:

Figure 36


17/24

The answer to the question above is 2: Script Task 3 will execute after either Script Task 4 or ScriptTask 2 executes and succeeds.

Is your head hurting yet? I hope not, because theres more! Lets next look at mixing constraints andexpressions.

Mixing Constraint and Expression Evaluation Operations

Stop the BIDS debugger if it is still running. Double-click the precedence constraint between ScriptTask 4 and Script Task 3 to open the Precedence Constraint Editor. Change the multiple constraintsoption to Logical AND. Set the Evaluation Operation to Expression and Constraint. Make sure theValue is set to Success and the Expression is set to @MyBool as shown in Figure 37:

Figure 37

In order for this precedence constraint to evaluate, the preceding task must execute and succeed, and

the value of the SSIS variable User::MyBool must be True. Lets look at the Variables (click the SSISdropdown menu from the top of the BIDS window and click Variables). According to Figure 38, MyBoolis set to False:


18/24

Figure 38

Will this precedence constraint ever evaluate to True? No. Why not? Because the Constraint(Success) must evaluate to True andthen Expression (MyBool) must also evaluate to True. ButMyBool is False. So it doesnt matter if Script Task 4 succeeds or fails, MyBool being False will preventthis precedence constraint from ever evaluating.

Try it. Execute the package in the BIDS debugger and observe the results. You will never see thepackage thus configured execute Script Task 3. It will always appear as shown in Figure 39:

Figure 39

Stop the BIDS debugger. Lets again open the Precedence Constraint Editor for the precedenceconstraint connecting Script Task 4 to Script Task 3. This time, lets change the Evaluation Operationto Expression or Constraint as shown in Figure 40:

Figure 40

Close the Precedence Constraint Editor and execute the package in the BIDS debugger. This time,Script Task 3 executes as shown in Figure 41:


19/24

Figure 41

Script Task 3 executes because both precedence constraints connected to it evaluated True. Its easyto understand how the precedence constraint between Script Task 2 and Script Task 3 evaluated. Buthow did the precedence constraint between Script Task 4 and Script Task 3 evaluate? The evaluationoperation is set to Expression or Constraint which means that either the expression or the constraintmust evaluate to True. If both evaluate to True, the precedence constraint will also evaluate to True.But the OR condition requires at least one either the expression or the constraint to evaluate True.The expression, MyBool, does not evaluate to True because the variable value is set to False. Theconstraint Successful execution of the preceding task evaluates to True. Thats why Script Task 3is allowed to execute.

Precedence Constraint Annotation

One thing you may have noticed: When we made the last change in the Evaluation Operation fromExpression and Constraint to Expression or Constraint there was no visual change in thegraphical representation on the Control Flow. This bugs me. How can I tell the difference? There is away: Click on the precedence constraint to select it and then press the F4 key to display Properties.The very first property is ShowAnnotation which can be set to ConstraintOptions as shown in Figure42:


20/24

Figure 42

The ConstraintOptions setting of the ShowAnnotation property displays text that clarifies the evaluationoperation of the precedence constraint as shown in Figure 43:

Figure 43

Mixing It Up!

First, lets edit the code in Public Sub Main for Script Task 2 and Script Task 4 to match whats listed in

Listing 2:

PublicSubMain()

DimsTaskName AsString=

Dts.Variables("TaskName").Value.ToString

DimiResponse AsInteger

iResponse =MsgBox("Succeed "&sTaskName &"?",_

MsgBoxStyle.YesNo +MsgBoxStyle.Question,

_


21/24

sTaskName &" Success Question")

IfiResponse =vbYes Then

Dts.TaskResult =ScriptResults.Success

Else

Dts.TaskResult =ScriptResults.Failure

EndIf

EndSub

Listing 2

This code creates prompts for task success from each of these script tasks. Edit the precedenceconstraint between Script Task 4 and Script Task 3, setting the Evaluation Operation to Expressionand Constraint, the Value to Failure, and the Expression to @MyBool as shown in Figure 44:

Figure 44

Edit the precedence constraint between Script Task 2 and Script Task 3, setting the EvaluationOperation to Expression and the Expression to !@MyBool as shown in Figure 45:

Figure 45

If the ShowAnnotation property of both precedence constraints is set to ConstraintOptions, yourControl Flow should appear similar to that shown in Figure 46:


22/24

Figure 46

Note that setting the Evaluation Operation to Expression causes the ConstraintOptions to displayCompletion and . As I shared earlier in this article, this matches the logical operation.

What will need to happen before Script Task 3 will execute?


Try it. One variation is to fail Script Task 4 and succeed Script Task 2 as shown in Figure 47:

Figure 47

This is a trick question. The answer is 4 Script Task 3 will never execute. Why? Because bothprecedence constraints are configured for Multiple Constraints Logical AND. They must both evaluatebefore Script Task 3 will be allowed to execute. But they contain a mutually exclusive condition withrespect to the value of the MyBool SSIS variable: one constraint evaluates when MyBool is True; theother when MyBool is False. MyBool will never be both True andFalse not at the same time (at leastnot in this universe). The lesson here: If mutually exclusive conditions exist, you must configure theMultiple Constraints to Logical OR at the task where the workflow converges.


23/24

Lets make it possible for this configuration to run by making a change to the precedence constraintbetween Script Task 2 and Script Task 3. Change the Expression to read @MyBool as shown inFigure 48:

Figure 48

Change the value of the SSIS variable MyBool from False to True as shown in Figure 49:

Figure 49

If I execute the package in the BIDS debugger, fail Script Task 4, and succeed (or fail) Script Task 2;Script Task 3 executes as shown in Figure 50:

Figure 50

Once we remove the mutually exclusive conditions, multiple constraints configured for Logical ANDcon evaluate and fire the next task.

Another Gotcha

Earlier we disabled the Sequence Container. Lets now disable Script Task 4 as shown in Figure 51:


24/24

Figure 51

When we execute the package in the BIDS debugger, Script Task 4 will not execute. But neither willScript Task 3 as shown in Figure 52:

Figure 52

Why not? MyBool is True and Script Task 2 completed. Script Task 4 is skipped what gives? WhenScript Task 4 is disabled, the Control Flow assertsSuccess. That means the Failure precedenceconstraint will not (ever) evaluate. Kudos to my friend Doreena Doherty for bringing this tidbit to myattention.

Conclusion

In this article, we used an SSIS variable to control precedence constraint evaluation, examined

Multiple Constraints, and took a look at a couple gotchas to avoid when configuring precedenceconstraints to control workflow in an SSIS package.

This article is part of the Stairway to Integration ServicesStairway

Level 8 Stairway to Integration Services

Documents

Transcript of Level 8 Stairway to Integration Services