© A

BB

Ltd

. -

1 -

NM DMS Incremental Update

Training

ABB

Annual Users’ Group Meeting

August 27 – 31, 2006

© A

BB

Inc

. -

2 -

IU Basics

Update the as-operated model with the as-built model

Designed to preserve database integrity for any type of failure

As-Operated changes(temporary states) are never lost

Production as-operated

model

GIS extract

as-built model

© A

BB

Inc

. -

3 -

IU Basics

Compare

Update

Rollback

Complete

© A

BB

Inc

. -

4 -

IU Basics - Compare

Four methods are available

Full - Comparison uses a full GIS extract

Sub_id – Comparison uses a subset of the full GIS extract Most common

Transactions (add/delete) – relies on GIS to provide all add/delete transactions

Transactions (add/delete/update) – relies on GIS to provide all add/delete/update transactions Simplest, in terms of processing, but only those customers using

ABB Data Maintenance tool can use this method

© A

BB

Inc

. -

5 -

IU Basics - Compare

All IU processing, but especially the Compare stage, is dependent on the ABB_INT_ID columns

These values are the link between the GIS system and the NM-DMS database

Values should never be reused Actually, they can, but must be careful

Values must be unique within a table, but can be duplicated in different tables

ie. Device.abb_int_id values must all be unique, but they can overlap values in the line table.

© A

BB

Inc

. -

6 -

IU Basics – ABB_INT_IDs

Typical detection of an Add transaction Abb_int_id value 10 exists in the GIS extract, but not in the

master

Therefore, it needs to be added to the master

Special care is needed when writing the GIS extraction tool

Often, there is not an exact one to one match between GIS and NM-DMS objects

ABB_INT_IDs must be generated consistently between extracts

© A

BB

Inc

. -

7 -

IU Basics – ABB_INT_IDs

Typical detection of an Add transaction Abb_int_id value 10 exists in the GIS extract, but not in the

master

Therefore, it needs to be added to the master

Special care is needed when writing the GIS extraction tool

Often, there is not an exact one to one match between GIS and NM-DMS objects

ABB_INT_IDs must be generated consistently between extracts

© A

BB

Inc

. -

8 -

1

2

34 5

GIS

1

2

34 5

node

nodeNode/line

NM-DMS

IU Basics – ABB_INT_IDs

© A

BB

Inc

. -

9 -

IU Compare - Full

1 2 3 4 1 3 4

5 6 7

Master CSF

© A

BB

Inc

. -

10 -

IU Compare – SUB ID

1 2 3 4

5 6 7 8

Master CSF

8 91 3 4 10

© A

BB

Inc

. -

11 -

IU Compare – SUB ID

1 2 3 4

5 6 7 8

Master CSF

8 91 3 4 10

8 12

© A

BB

Inc

. -

12 -

IU Compare – SUB ID

Only compare against what is in the CSF

If you delete an entire SUB_ID value, the extractor must create an entry in the AFFECTED_SUBID table Rare!

Boundary objects – Tie points, may be extracted for feeder 1 or feeder 2

Objects can move between sub_id sets, via updates Again, MUST be extracted consistently

© A

BB

Inc

. -

13 -

IU Compare – Add/Delete transactions

Add/Delete transactions are explicitly specified in the CSF.csf_trans table

No detection of add/deletes is done by the compare process

Updates are still detected Including link changes

Including path points

© A

BB

Inc

. -

14 -

IU Compare – Add/Delete/Update transactions

Add/Delete/Update transactions are explicitly specified in the CSF.csf_trans table

No detection of add/deletes/updates is done by the compare process

Updates are still detected Including link changes

© A

BB

Inc

. -

15 -

IU Compare – Path Points

Path Points require special IU rules, as individual path points do not have abb_int_id values.

Brute force comparison during line update detection

For each line Each set of path points, both geographic and schematic are

compared

Any difference in X,Y coordinates, or in number of path points, trigger an update

The update is the complete replacement of the list of path points

© A

BB

Inc

. -

16 -

IU Compare – Transactions Types

Four transaction types

Add

Delete

Update

Temporary ID Change

Link Change – now modeled as a delete/add pair of transactions

© A

BB

Inc

. -

17 -

IU Compare – Link Changes

Link changes are changes in internal object dependencies

Ex. Device -> Line

Line -> Node 1

Line -> Node 2

Load -> Node

Source -> Node

Capacitor -> Node

Mpoint -> Node

Site -> Node

© A

BB

Inc

. -

18 -

IU Compare – Link Changes

Special case for lines

If a line has a node link change(either node 1 or 2)

A delete/add pair is created for the line

A delete/add pair is also created for each device on the line

© A

BB

Inc

. -

19 -

IU Compare – Circular ID changes

Object character IDs must remain unique

Problems occur when IDs are swapped Device ID #1 is changed to Device ID #2

Device ID #2 is changed to Device ID #1

Cannot be performed directly, as it will violate unique restriction

This “Circular” dependency is detected, and a temporary ID transaction is injected Device ID #1 is changed to dummy value

Device ID #2 is changed to Device ID #1

Device ID #1 is changed to Device ID #2

No unique ID violation

© A

BB

Inc

. -

20 -

IU Compare – Updates

Objects are matched using their abb_int_id values

Columns are compared, based on csf_column_comp table

© A

BB

Inc

. -

21 -

IU Compare – Security Checks

Delete object with Tag

Delete node with lines/jumpers attached

Delete node with temporary generator

Delete line with line cut attached

Delete node/line with phase jumper attached

Delete device, load, source that is part of current outage

Unique ID reference No options available to prevent unique ID violation

© A

BB

Inc

. -

22 -

IU Update - Basics

Updates are done atomically – so that model is not corrupted in the event of database or hardware failure

Updates also save all necessary data for the rollback stage

© A

BB

Inc

. -

23 -

IU Update – Order of updates

Check SHM for available space Ways to increase space

ADM_DATABASE.SHMBUF

Manipulate XXX_FREE tables

Deletions

Additions

Updates

© A

BB

Inc

. -

24 -

IU Update – Order of deletions

Node objects

Line device

Line

Node

Memo

Feeder

Substation

ATO

Node objects Load

Capacitor

Site

Source

Mpoint

© A

BB

Inc

. -

25 -

IU Update – Order of additions

Node

Line

ATO

Line device

Node objects

Memo

Feeder

Substation

Node objects Load

Capacitor

Site

Source

Mpoint

© A

BB

Inc

. -

26 -

IU Update – Order of updates

Line device

Path Point

Line

Node objects

Node

Memo

Feeder

Substation

ATO

Node objects Load

Capacitor

Site

Source

Mpoint

© A

BB

Inc

. -

27 -

Typical Delete transaction processing

Create csf_reversal record

Create CHG_XXXXX record

Delete from base table

Delete from memory

Send message

© A

BB

Inc

. -

28 -

Typical Add transaction processing

Create csf_reversal record

Insert into base table, after getting next free fpos

Lookup any reference fpos values (link ids)

Insert into memory

Update csf_reversal record with new fpos

Send message

NO CHG_xxx table entry for additions

© A

BB

Inc

. -

29 -

Add transaction processing, for a link change

Steps are as for a regular add

Additional step to lookup the saved status value in the CHG_XXX table Stored there during the delete

© A

BB

Inc

. -

30 -

Looking up link ids

FPOS values vary between CSF and master

Example – Adding line with abb_int_id 50

It has no_key_1, no_key_2 references of 15,20 in CSF

These are the FPOS values, not abb_int_id values

Must query CSF to determine abb_int_id value of node where fpos = 15, 20

Then, when inserting into master db, must use these node abb_int_id values to lookup the correct fpos value

© A

BB

Inc

. -

31 -

Looking up link ids

CSF

Line – fpos 30

abb_int_id 50

no_key_1 10

no_key_2 15

Node – fpos 10

abb_int_id 100

Node – fpos 15

abb_int_id 101

Line – fpos 99

abb_int_id 50

no_key_1 1200

no_key_2 1501

Node – fpos 1200

abb_int_id 100

Node – fpos 1501

abb_int_id 101

Master

© A

BB

Inc

. -

32 -

Looking up link ids

The FPOS to abb_int_id translation is also performed when creating the CHG_XXX table entries

The CHG_XXX tables are exact copies of their matching object tables, with some additional columns

These columns support the fpos-abb_int_id lookup values, and sequencing of updates

Common cause of failure during the update process

© A

BB

Inc

. -

33 -

Typical Update transaction processing

Create csf_reversal record

Create chg_XXX record

Lookup up record from CSF

Update fields in master db

Update in memory

Send message

Path points are slightly different. All path points for a line (geo/schem) are replaced, even if only one

has changed

© A

BB

Inc

. -

34 -

Updates – temporary states

Temporary (as-operated) status is not modified

EXCEPT FOR

When built phases change, closed phases are adjusted accordingly

Ex.

Device is built ABC, closed ABC

Built phases are changed to BC

Closed phases are modified to BC

© A

BB

Inc

. -

35 -

Rollback

Simply reverses all applied transactions

Update processing, in reverse

Based on csf_reversal table

© A

BB

Inc

. -

36 -

Rollback – Order of transactions

Insertions

Deletions

Updates

© A

BB

Inc

. -

37 -

Rollback – Reversing insertions, order

Line device

Line

Node objects

Node

ATO

Memo

Feeder

Substation

Node objects Load

Capacitor

Site

Source

Mpoint

© A

BB

Inc

. -

38 -

Rollback – Reversing deletions, order

Node

Node objects

Line

ATO

Line device

Memo

Feeder

Substation

Node objects Load

Capacitor

Site

Source

Mpoint

© A

BB

Inc

. -

39 -

Rollback – Reversing updates, order

Node objects Load

Capacitor

Site

Source

Mpoint

Line device

PathPoints

Line

Node objects

Node

ATO

Memo

Feeder

Substation

© A

BB

Inc

. -

40 -

Complete

Cleanup all records needed for rollback Csf_reversal

Chg_xxxx

Modifies CSF status in ADM_DATABASE table

© A

BB

Inc

. -

41 -

ADM_DATABASE Status values

1 – Indicates a master database

2 – A valid case study, ready for compare

3 – Casestudy, but corrupted – failed on creation

4 – Update attempt failed

5 – Rollback failed

6 – Successful completed

7 – Casestudy, but not valid for the IU process

8 – Update successful

© A

BB

Inc

. -

42 -

External script hooks

Each script takes arguments of Astatus id, astatus password, astatus dbname(TNS connect)

Scripts are references using the environment variables below

PRE_DIFF_CHECK_SCRIPT Called before performing the Compare

POST_DIFF_CHECK_SCRIPT Called after the Compare completes

POST_INCR_SCRIPT_NON_COMPLETE Called after Update is complete

POST_ROLLBACK_SCRIPT Called after Rollback is complete

POST_INCR_SCRIPT

Called after Complete is complete

© A

BB

Inc

. -

43 -

External script hooks

Why use these?

Auto restart outageEngine after update or rollback

Auto run nodeLoc program, to keep node_location table correctly populated

Update customer counts on loads

Any other processing you need

© A

BB

Inc

. -

44 -

Some options

PathPoints in memory

If enough memory is available, read all the CSF path points into memory at once, instead of line by line during the compare

Cut roughly 25% off Chicago compare times

Lookup Link IDs As discussed before, many database queries are needed

to lookup abb_int_id link values from the CSF during the update

If the abb_int_id values(In the CSF only) are used in place of the minimized FPOS values, these lookup steps can simply be skipped.

Still able to load the CSF for viewing using the GUI

Not able to run opmanager, outageEngine, server apps

© A

BB

Inc

. -

45 -

Some options

FeederSub compare and update

If set, include feeder and substation tables in the IU process

Can use sub_id or full method

The ID column is both tables is treated as the abb_int_id value

© A

BB

Inc

. -

46 -

IU Troubleshooting

The previous slides contain everything you need to know to immediately troubleshoot any IU problems

HEHEHE

IU is one of the most complicated processes we have

© A

BB

Inc

. -

47 -

IU Troubleshooting

So it failed, now what do you do?

First, check the error log, usually located in the /tmp directory Btw, this location can be changed by using the TMPDIR

env variable

Also, check the output of the incrUpd process itself

© A

BB

Inc

. -

48 -

IU Troubleshooting – Common errors

Corrupt data in the master database

Fails when trying to create chg_xxx records

Ex.

Device -> li_key is 0, or points to a non-existent line

When trying to lookup the abb_int_id value to save in the chg_xxx table, failure

© A

BB

Inc

. -

49 -

IU Troubleshooting – Common errors

Solution

Simply delete the offending object

Because the link data is bad, its not loaded into the GUI, or the server SHM

Once deleted, its detected as an ADD transaction

Detection of these problems can be done before hand using the pre diff check script

© A

BB

Inc

. -

50 -

IU Troubleshooting – Common errors

Tag added after compare stage

Jumper, generator added after compare stage

Unique ID violations, temporary objects

© A

BB

Inc

. -

51 -

IU Troubleshooting – Steps

If the update fails, you must either complete, keeping what you have successfully updated

Or rollback, which is recommended

You cannot try to resume the update where it left off

Fix the problem, recompare and retry update

© A

BB

Inc

. -

52 -

IU Troubleshooting – Steps

If the rollback fails, you can correct the problem, then simply re-rollback

Typical cause of rollback failure

Object is being added Csf_reversal record created

Some next step fails – unique id, db error

Csf_reversal record remains

When reversing, this addition must deleted, but since it doesn’t exist, the rollback fails with an error

Simply delete the csf_reversal record and re-rollback

© A

BB

Inc

. -

53 -

IU Troubleshooting – Strange behavior

Compare, Update, then Re-compare

Additional updates are found

Typically because of how null values are treated during the update.

Numeric columns typically receive a value of 0

Character columns typically receive a value of ‘ ‘ That’s a space

During the re-compare, they show up as updates

© A

BB

Inc

. -

54 -

IU Troubleshooting – Strange behavior

Link changes – delete/add pairs

Operators may try to operate the devices after they have been deleted, but before they are re-added

Results in a server error

Usually, there’s no contention when trying to operate an object flagged for deletion, but the new processing for link changes introduces the possibility

© A

BB

Inc

. -

55 -

IU interaction with the GUI

As all the changes are occurring on the server, the GUI receives each message and holds onto it Indicator at bottom of GUI console shows message count

After IU is complete(a complete message is sent), the GUI will process all the messages

Can take a while

GUI user pref to control processing, or ask the operator to logout/in