Software Group | IBM Israel Software Laboratories SOA ...

Software Group | IBM Israel Software Laboratories SOA Advanced Technologies

© 2007 IBM Corporation

Joshua FoxRegulatory Compliance through Metadata Mining

Software Group | Israel Software Labs


What Does My IT System Mean?

Real World

Metadata



Use Case: Security Marking A simplified example Security labeling has many drivers Focusing here on the semantics



Weaponization-related


Use Case: Security Marking

Not Weaponization-

related

Not Weaponization-

related

Not Weaponization-

related

Not Weaponization-

related

Not Weaponization-

related

Not Weaponization-

related

Not Weaponization-

related

Not Weaponization-

related




Biotech Lab

A lab takes its first DoD contract Needs DIACAP approval; cannot risk non-compliance Needs to apply security markings for access control in the

Information Sharing Environment



The Metadata

Metadata for structured (machine-read) data Database schemas Web service WSDLs COBOL copybooks UML & DoDAF Models



Security Markings: Find Subject Find all info services in semantic

area of, e.g. “weaponization” Metadata Repository holds service

descriptions, database schemas, other metadata

Repository also holds standard categories from data dictionary

Tool proposes categorization Analyst uses this as input, saving

valuable manual-analysis time

Semantics

Metadata



<>…<>

<>…<>

<>…<>

Historical MD Situation MD in small quantities Scattered in

DBA teams Development teams



Background

Trends in leading-edge enterprises

Large,

cross-organization,

metadata repositories



The Promise:

Governance across the organization,

but…



Mess of Metadata

<xsd>

…

<xsd>

<xsd>

<xsd>

……

<xsd>

<xsd>

<xsd>

<xsd>

<xsd>

……

……

<xsd>



Heterogeneity in Metadata Different technologies: XML,

RDB, UML Different structures and

terminologies

<xsd>…

<xsd>

<xsd>

<xsd>

……

<xsd>

<xsd>

<xsd>

<xsd>

<xsd>…

…

……



Confused Semantics in Metadata

Tank?

Army

Navy



Confused Semantics in Metadata

“Secure” NSA: No eavesdropping Air Force: Buy it Army: Guard the perimeter Marines: Storm it Navy: Lock the door, turn

off the lights



Huge Quantities of Metadata

<xsd>…

<xsd>

<xsd>

<xsd>

……

<xsd>

<xsd>

<xsd>

<xsd>

<xsd>…

…

……



Older Approaches

Build taxonomy/ontology Map it to the metadata

Metadata (e.g., XSD)

Ontology



Older Approaches Don’t Work



Older Approaches Don’t Work Painstaking human labor



Older Approaches Don’t Work Painstaking human labor High-cost labor: IT+ business

knowledge

$$

$

$



Older Approaches Don’t Work Painstaking human labor High-cost labor: IT+ business

knowledge: Consultants!

$$

$

$



Older Approaches Don’t Work Painstaking human labor High-cost labor with IT+ business

knowledge: Consultants! Beyond human limits

$$

$

$

:-(

:-(

:-(

:-(

:-(



New Opportunities Created By:

Moore’s Law Great progress in Data Mining

Searching, classifying and organizing

Recent innovative uses: Terrorist Threat Analysis Security, Web 2.0, Google



The Time is Right

Well-known search and information-management techniques

Now, apply them to metadata



Compliance

MetadataRepository

Functional Architecture

Persistence

Semi-automation ofmapping

Engine

BusinessFunctionality Access Reporting

Real-LifeMeaning

Ontology(AKA taxonomy, dictionary,

glossary, logical model, categories)

Mapping(ontology <->metadata)



Methodology

1) Prepare Metadata2) Set up Categories3) Machine Learning 4) Suggest Category



(1) Prepare Metadata

a) Load metadata into repository

b) Pre-process metadata into Text: e.g., “Deployment”, “Location” Structure: e.g., “Deployment:Location” to

represent Table and Column



(2) Set up Categories

(AKA taxonomy, ontology, glossary, data dictionary, business model, domain model)

a) Follow Security Classification Guide

b) May use Community-of-Interest (CoI) vocabulary

c) Defense Discovery Metadata Standard for categories

d) Keep it simple!



(3) Machine Learning

a) Training on a sample of metadata samples

b) Provide semantic category mappings for this sample

c) Standard Bayesian classification algorithms learn common or uncommon words in a category



(4) Suggest Category for Metadata Item

a) Preprocess metadata

b) Submit to classification engine

c) Receive suggested category

d) Proceed with analysis

Cla

ssificatio

nE

ng

ine

Metadata

Analyst

Humans and machines complementing each other



Understand Your IT: Use Cases Legacy Transformation: What business services are

hiding in your legacy applications? Reuse: Where is a service with this business

functionality? Fast Start for Community of Interest



Non-Financial

Non-Financial

Non-Financial

Non-FinancialNon-Financial

Non-Financial

Non-Financial

Non-Financial

Financial

Financial

Non-Financial

Use Case: SOX Reporting



SOX ComplianceReal World

Metadata

A Telco needs to comply with SOX to avoid penalties

Build reports from all info services with “financial” information

Metadata repository holds services, DB schemas, etc.

Tool proposes categorization Analyst can find relevant data sources

more quickly, then build report



Why Mine the Metadata

Services: Invocation-level data is transient

Metadata already expresses semantics of the data

Metadata uncoupled from ever-changing data

Table: Troop_ Deployment

Column: Total

Troop_Deployment

… … … Total

154,650

25,390



Mining the Metadata: More Secure Tool & human analyst do

not access actual data Human analyst can avoid

accessing even the metadata

Table: Troop_ Deployment

Column: Total

Troop_Deployment

… … … Total

154,650

25,390



Data Mining Complements metadata

mining Build metadata from data Differentiate on the

resource level

Table: Deployment

Column: Location

Deployment

… … … Location

“DC LAN 1”

“Baghdad LAN 2”



SimplicityMetadata Data

Structured Data Documents

Coarse-Grained Fine-Grained

Classification Search, metadata-internal relationships, transformation-building

Schema-to-Semantics Schema-to-Schema

Feasible Long-term Research

Reusable Functionality Specialized Functionality

Business Value Technical Value

Our focus



SummaryReal World

Metadata

Too much metadata: humans need help Use your metadata repository Understand your metadata Identify relevant metadata Comply with regulations using IT

metadata Metadata mining: The time is right



Joshua Fox

Metadata Analytics

Israel Software Labs

IBM

[email protected]

http://www.joshuafox.com

Thank you

Software Group | IBM Israel Software Laboratories SOA ...

Documents

Transcript of Software Group | IBM Israel Software Laboratories SOA ...