Post on 05-Aug-2015
© 2015 IBM Corporation
IBM InfoSphere Information Server Stewardship Center and Data Quality Exception Console
2 © 2015 IBM Corporation
Agenda
IBM Data StewardShip Center
Data Quality Exception Console
Questions & Réponses
© 2015 IBM Corporation
Stewardship Center
5 © 2015 IBM Corporation
Governance Officers
Data Stewards (IT and LOB)
Data Analysts/SMEs
Web
Mobile
Social Collaboration
DashboardWorkflow &
Rules
Provide a single collaborative environment for business users to
define and monitor compliance with data quality policies and to
manage data quality issues to resolution
Addressing the needs of various roles across the governance organization
Key capabilities delivered within customizable and collaborative interfaces
IBM InfoSphere Stewardship Center
6 © 2015 IBM Corporation
InfoSphere Stewardship Center: Event Notification
with Workflow and Remediation
Read File
Errors
Business Term
Updates
Notifications/Email
Exception
Fixes
Quality Exception
NotificationDuplicate
Resolution
Policy
Changes
Suspect
Duplicates
Catalog Updates
7 © 2015 IBM Corporation
Data StewardStewie Silvers
Governance Council Member
Gary Grayson
Data Steward Manager
Martin McGregor
Meets the Needs of Critical Business Roles
Traditional Data Steward and LOB users
collaborate to remediate record violations,
improving data quality.
Data Steward Manager responsible for overseeing a team of subject matters experts.
Data Quality Champion and LOB leader, collaborate defining data policies and rules for the business in addition to monitoring policy compliance
8 © 2015 IBM Corporation
Stewardship Center Performance Dashboard
The Stewardship Center provides out of the box Dashboards:– Team Performance and Process Performance
9 © 2015 IBM Corporation
Stewardship Center takes advantage of IBM BPM
Enterprise Processes
Services
Information Server Services
CustomerPatientVendorProductAccount
IBM BPM & Decisions
BPM Standard
OnboardingRegistrationParty updates
SalesMarketingDelivery
Information
Governance and
Data Quality
Stewardship
Information Server and IBM Business
Process Management come together to
help you manage your Governance
and Data Quality processes:
Information Server delivers real time data
exceptions and events to be governed using
a repeatable business process
BPM provides orchestration of the steps
required to improve data quality and increase
common understanding
Business Process Management is a discipline that leverages software and
services to provide total visibility into your organization. Discover, document,
automate, and continuously improve business processes to increase efficiency
and reduce costs.
10 © 2015 IBM Corporation
Go Mobile, Get Social, and Govern
Mobile
Access
Social Collaboration
Governance
Workflow
Reporting
11 © 2015 IBM Corporation
What is the Stewardship Center?
The integration of IBM BPM Standard Edition and Information Server– ‘Stewardship Center’ is a customized home screen for IBM BPM Process Portal
IBM provides a set of sample BPM applications to demonstrate the
basic capabilities– Exception Management
– Information Governance Catalog Governance Rule workflow
– Information Governance Catalog Business Term emails
This section will focus on the Exception Management
sample application– Information Governance Catalog applications have already been discussed
12 © 2015 IBM Corporation
InfoSphere Stewardship Center
Single Inbox and Remediation Portal
MDM tasks
DataStage/
QualityStage
tasks
Information
Analyzer
tasks
Information
Governance
Catalog tasks
13 © 2015 IBM Corporation
Start
Exception list
submit
fixes
reject
approve
Approval
request list
Apply fix request list
escalate
apply fix
Data Steward
Data Steward Manager
Data Source Owner
End
Out Of the Box Exception Remediation Sample
Endresolve
User=silvers
Name=Stewie Silvers
User=mcgregor
Name=Martin McGregor
User=williams
Name=Chad Williams
Exception
14 © 2015 IBM Corporation
A View in IBM Process Designer
15 © 2015 IBM Corporation
Workflow – Data Steward
Sign in as Data Steward ‘silvers’
See ‘Open Tasks’ new exceptions
Sample application as defined
the due date as date run + 24
hours.
PoC/Demo Tip:
Switch between ‘Open Tasks’
and ‘Completed Tasks’ to
refresh the screen
PoC/Demo Tip: Carefully read to see the exception source
16 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Claim the task
17 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Work with the task
PoC/Demo Tip:
‘Submit’ after changes
‘Escalate’ if you don’t know what changes to make
18 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Work with the task (continued)
View the BPM
Process Diagram
19 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Work with the task (continued)– See the exception description
You can drill directly to the Information
Governance Catalog to see the Data Rule
The link is AUTOMATIC and illustrates integration of our suite!
20 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Work with the task (continued)– See the exception description definition in IGC
The Data Steward can find what they need with a few clicks!
…and continue to drill into
the Data Rule Definition
21 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Work with the task (continued)– Edit and change individual records
The sample application allows any data to be changed.
It also shows how it is possible to have a ‘read only’ flag
to prevent data change.
The sample application only allows
editing of one exception at a time. Stewardship Center supports EDIT!
22 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Work with the task (continued)– Edit and change individual records (continued)
23 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Work with the task (continued)– View changed records
Changed records are highlighted in light blue.
Changed data is highlighted in dark blue.
Data
Steward can
VIEW
changes!
24 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Work with the task (continued)– Changing multiple records with Export
• Also can be used to share data with others
25 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Work with the task (continued)– Changing multiple records with Export (continued)
A csv file is downloaded to the client
Every column output by the rule will have a ‘_new’ column
26 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Work with the task (continued)– Changing multiple records with Export (continued)
You can use Microsoft™ Excel® to edit the file
PoC Tip:
Be very careful when opening .csv files in Excel.
Any data with leading zeroes will have those
leading zeroes remove. For example, the valid US
Postal Code 02201 will become the invalid US
Postal Code of 2201.
Excel will also corrupt timestamps: 2015-01-25
09:40:13.0 becomes 40:13.0.
27 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Work with the task (continued)– Changing multiple records with Export (continued)
Then ‘Import’
Your Firefox ‘import’ screen will be blank
PoC/Demo Tip:
Import does not currently work with
Internet Explorer. Use Firefox instead.
28 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Work with the task (continued)– Changing multiple records with Export (continued)
You will need to change the number of ‘Rows per page’ to force a
refresh
29 © 2015 IBM Corporation
But Where Is The Fixed Data Stored?
The fixed data is stored in the providers repositoryDataStage and QualityStage Exceptions Stage = Exceptions Stage Database
30 © 2015 IBM Corporation
But Where Is The Fixed Data Stored? (continued)
DataStage and QualityStage Exceptions Stage– In the same output table as the exceptions
• Reminder: Get the table name from the job run:
31 © 2015 IBM Corporation
But Where Is The Fixed Data Stored? (continued)
DataStage and QualityStage Exceptions Stage (continued)– In the same output table as the exceptions (continued)
– The data is defined as a BLOB
The data is JSON stored in binary format, but CAN
be parsed by the Hierarchical Stage…
This is the view from IBM Data Studio
© 2015 IBM Corporation
Back to the Workflow
33 © 2015 IBM Corporation
Workflow – Data Steward (continued)
Data Steward can do one of two things:– Submit Fixes
• When changes are done
– Escalate Task• Ask the Data Steward Manager to review unknown exceptions
Updates shown in the DQEC
34 © 2015 IBM Corporation
Workflow – Data Steward Manager
When the Data Steward selects ‘Escalate’– Resolve – the task is complete
– DQEC is updated
35 © 2015 IBM Corporation
Workflow – Data Steward Manager
(continued)
When the Data Steward selects ‘Submit Fixes’
– Approve – goes on to Data Source Owner
– Reject – goes back to Data Steward• Any fixes are also discarded
– Either one updates the Data Quality Exception Console
36 © 2015 IBM Corporation
Workflow – Data Source Owner
When the Data Source Owner select ‘Approve’– Apply Fixes
• Once changes done (however they are done)
– Data Quality Exception Console is updated
37 © 2015 IBM Corporation
Start
Exception list
submit
fixes
reject
approve
Approval
request list
Apply fix request list
escalate
apply fix
Data Steward
Data Steward Manager
Data Source Owner
End
Out Of the Box Exception Remediation Sample - Done
Endresolve
Exception
38 © 2015 IBM Corporation
InfoSphere Stewardship Center
CollaborationMonitoring
Inbox
Data Quality Exception ConsoleCommon
Event
Framework
Information Analyzer
MDM Server
DataStage/QualityStage
Exception Stage
Information Governance
Catalog
Information Governance Dashboard
API
(Information Analyzer,
DataStage/QualityStage
Exceptions Stage)
Source
Systems
WorkflowsWorkflows
1a 1b
2a
2b
1c
1d
3
1a-d
2a-b
3
Information Analyzer and DataStage/QualityStage
Exception Stage exception events and notifications
Information Governance Catalog events and notifications
MDM integration
InfoSphere Stewardship Center Architecture
Event Notifier
Event Registry
Event Service
39 © 2015 IBM Corporation
Subscriptions
Subscription Manager is on the Information Server Launchpad
40 © 2015 IBM Corporation
Subscriptions (continued) PoC/Demo Tip:
Name your subscriptions:
Asset Type + Application + Other Information
For example: Business Term Basic Email Notification Create
Email is available for all Event Types. Application will only
be listed if there is a registered Callback.
All subscriptions allow filtering.
Filters must be a single item.
ACTION=CREATE or
ACTION=MODIFY would require
two subscriptions.
41 © 2015 IBM Corporation
Subscriptions (continued)
Notification Type ‘Email’ available to all event types– Make sure to set up email provider (the smtp server) as per this link
– The email will go to the email address entered for the user
– You can send a copy to any other user
PoC/Demo Tip:
Subscription Manager Email only supports ‘no security’ (port
25), or ‘SSL’ security (port 465). TLS (port 587) is not
supported. Hotmail, Live, and Outlook all use TLS.
42 © 2015 IBM Corporation
Subscriptions (continued)
Notification Type ‘Email’ available to all event types (continued)– At GA, the email will look better for Remediation events than for IGC events
• Remediation DataStage/QualityStage emails:
• Information Analyzer emails:
• IGC emails (from creation of a new Category):
Work is underway to make the emails customizable.
43 © 2015 IBM Corporation
Subscriptions (continued)
Note the Event Types are not sorted
44 © 2015 IBM Corporation
Notifications
Notifications are published every time a subscribed action is
performed– Notifications are ONLY generated for subscribed actions
You can list and delete notifications based on time ranges– -beginDate “YYYY-MM-DD hh:mm:ss”
– -endDate “YYYY-MM-DD hh:mm:ss”
– You can specific just:• beginDate (everything from a specific time forward)
• endDate (everything up to a specific time)
© 2015 IBM Corporation
Thank You
End of Section 3
46 © 2015 IBM Corporation
Agenda
Stewardship Center Overview
Brief BPM Overview
High Level Architecture
Information Governance
Catalog– Basic Event Notification
– Sample IBM BPM Applications
Labs Part 1
Data Quality Remediation– Data Quality Exception Console
– Stewardship Center
Labs Part 2
Common Event Framework– Event Management Architecture
– Event Providers
– Event Types
– Callbacks
– Subscriptions
– Notifications
– Events
– Task Scheduling
Labs Part 3
Appendices– Delete from Data Quality
Exception Console
– Delete from Stewardship Center
Objective: Clearly Explain Stewardship Center Architecture,
Capabilities and Competitive Advantages
© 2015 IBM Corporation
Data Quality Execption Console
48 © 2015 IBM Corporation
Data Quality Exception Console
Replaces the Data Quality Console– Data Quality Console still available today
– Data Quality Console still fully functional
– Will be removed completely in a future release
Is the first step to managing exceptions in the Stewardship Center– Exceptions ‘sets’ must be selected to be ‘managed’ for the the out-of-the-box
application
The Data Quality Exception Console allows only critical exceptions
to be managed in the Stewardship Center.
49 © 2015 IBM Corporation
Data Quality Exception Console (DQEC)
Start from the Launchpad
50 © 2015 IBM Corporation
Data Quality Exception Console (continued)
Filters to limit what is displayedExceptions
Browser based Link to Information
Governance Dashboard
Sort on any column
51 © 2015 IBM Corporation
Filtering
Many Filter options
Combining filters is an ‘and’– Example: Contain string ‘val’ and produced by
Information Analyzer
Filters are predefined
Filters can not be saved (yet)
52 © 2015 IBM Corporation
What Does The Data Quality Exception
Console Show?
Information Analyzer– Output from Data Rule run
– Output from Rule Set run
This option does not matter.
Whatever you output will go to the
Data Quality Exception Console.
PoC Tip:
Choose wisely!
These are the columns seen
in the Stewardship Center
PoC Tip:
DO NOT use spaces in
your Data Rules Names!
53 © 2015 IBM Corporation
What Does The Data Quality Exception
Console Show? (continued)
DataStage and QualityStage– Output from Exceptions Stage only
• Not from the One-Source Match Exceptions nor the Two-Source Match Exceptions
Tip: Always include the ‘ruleID’
from the Data Rules Stage so
you know which Data Rule
Definition caused the exception
54 © 2015 IBM Corporation
What Does The Data Quality Exception
Console Show? (continued)
Information Analyzer Data Rule Definition and Rule Set Definition
‘Test’ runs will also show
System assigned Name
Poc/Demo Tip:
Delete these test runs when
testing is completed.
55 © 2015 IBM Corporation
Displaying Exception Details
Status will be updated by
the Stewardship Center!!!!!!!
Role based actions
For example: ‘Administrators’ can
only ‘Delete’
The Data Quality Exception Console is integrated!
56 © 2015 IBM Corporation
Viewing Records
More details to come…
Demo Tip:
Do not use the
browser ‘back’
button. Click
here instead.
57 © 2015 IBM Corporation
‘Edit’ and ‘Delete’
Edit to change the exception priority
– Not to edit the data (that can be done in the Stewardship Center!)– Not reflected in Stewardship Center– Is for this exception only, not all instances of this exception
58 © 2015 IBM Corporation
‘Edit’ and ‘Delete’ (continue)
Delete to remove the exception
– Also removes the output table• Does not matter what type – all are deleted
59 © 2015 IBM Corporation
But What About ‘Manage’ and ‘Monitor’?
All exceptions will be visible in the DQEC
Data Steward Manager decides how to handle the exceptions with:– Manage = Which exceptions are to be managed in the Stewardship Center
– Monitor = Which exceptions are to be monitored in the DQEC
60 © 2015 IBM Corporation
But What About ‘Manage’ and ‘Monitor’? (continued)
Selecting ‘Manage’ will make this exception visible in the
Stewardship Center– AND all future exceptions with the same name
61 © 2015 IBM Corporation
Questions & Réponses