InforSense KDE User Training Introduction to InforSense KDE and Building Workflows.

InforSense KDE User Training

Introduction to InforSense KDE and Building Workflows

© 2007 InforSense Ltd. 2

Agenda

KDE Introduction

What are workflows, nodes, and other terms

InforSense KDE Client (GUI tour)

Importing Data Sources

Building and Executing a Simple Workflow

InforSense KDE Portal

Introduction to Deployment

Case Study Exercise


Agenda

KDE Introduction







Case Study Exercise


What is InforSense KDE?

KDE = Knowledge Discovery Environment

Enterprise Software

Variety of applications

Range from Business Intelligence, Financial Solutions, Life Sciences and Health Sciences

Analytical workflows

Helps researchers and data analysts to access and integrate all their tools


What is the basic KDE Architecture?

KDE Core

Plugins = product extensions

KDE Server

RelationalDatabases

3rd PartyApplications

KDE Client(Build Workflows)

KDE Client(View Portal)

KDE Client(Analytics Tools)


Some Architecture Definitions Server: computer or a device that manages tasks on a

network resource

Client: pc or workstation that requests services from the server to perform some operations; it is where workflows are built and portal pages are viewed

Portal: web based application where end users can use the functionality created by workflows hidden behind a customized GUI

Plug-in: related group of functionalities (typically derived from multiple components/nodes) that can be added to the InforSense platform

Product: one or more plug-ins grouped in a domain area for purchase


Product List - Extensions InforSense BioSense

Specialized extension for bioinformatics

InforSense GenSense A solution for analyzing SNP chips

InforSense ChemSense Specialized extension for cheminformatics

InforSense TextSense Specialized extension for literature analytics and text mining


Product List - Extensions

InforSense Developer Pack

Comprises of a feature-rich Software Development Kit (SDK) and Application Programming Interface (API)

Enables users to tailor the standard InforSense KDE and IOE environments for their specific requirements


Agenda

KDE Introduction







Case Study Exercise


What is a Workflow? InforSense KDE uses a workflow paradigm to perform

operations on different data

Components in a project panel must be connected together in to a workflow in order to carry out a task

The direction of the arrows connecting the nodes indicates the logical flow of objects, such as data tables and models, during each stage of analysis

Execution Sequence

Imported Data Operation 2

Result Data NOperation 1 Operation n

Execution 1

Result

Data 1

Result

Data 2

Execution N


What is a Workflow?

In KDE, all the components of your analysis (data sources, operations and results) are represented as icons

The interactions between the components are represented by using directed arrows, linking the components together


What is a Component or Node? An individual unit of functionality that performs a

task on the data

Typically (but not always), has input(s), output(s), and parameters to be decided by the user

Input Ports

Input Ports

Output Ports

Output Ports ParametersParameters


Other KDE Terms

Table: Most nodes process data in a table format with rows, columns, and each individual cell contains information

Model: Some nodes can build or use models that predict results based on a set of data

Inputs: table or model given to a node to perform an operation; there may be 0, 1, or multiple inputs for any node; inputs can be required or optional

Outputs: table or model produced by a node from an operation; there may be 0, 1, or multiple outputs for any node

Parameters: Variables whose values can be entered by the user just prior to running a workflow


More KDE Terms Ports: iconic representation of the inputs and outputs for a

node; if there is more than one port involved in a connection between nodes, the user will need to select the correct one

Metadata: KDE nodes generally keep track of the number and names of the columns in the data table in order to disallow workflow connections that do not make logical sense based on the metadata

Execute: Performing the operation of a node and all nodes preceeding it

Cache: for normal processes, KDE stores the table in a temporary table once it has been executed; this makes it faster to retrieve if the node needs to be re-executed, but if parameter values change then the cache needs to be cleared to remove old values


Agenda

KDE Introduction







Case Study Exercise


Logging into the KDE Client The server must be started before the client can be

launched

The workflow building client tool can be launched from:

The KDE portal (webstart is recommended)

From a desktop icon or item in your Start menu

Ask your administrator for the web address - it is usually <ServerIPaddress>:port number/kweb


Login Screen

After the client is downloaded, you will be asked for authentication Administrators would typically manage username/passwords - here you can use the default account: demo/demo

1

2


Client Interface - Overview

Context Sensitive Toolbar


ResourcesPanel

ResourcesPanel

UserspaceUserspace

Components

Components

Task Manager

Task Manager

Navigator View

Navigator View

Project Panel

Project Panel

Node Editor

Node Editor

The client interface is where you construct workflows



The toolbar provides shortcut access to the most commonly used functionalities Some of the toolbar buttons are also context sensitive, and will enable/disable depending on what is currently selected



Many icons are standard software icons List of icons specific to KDE are below:

No Icon Shortcut to No Icon Shortcut to

01 Search 05 Node Help

02 Mount Userspace 06 Main help

03 Unmount Userspace

07 Execute

04 Refresh 08 Available tools


Resources Panel The Resources panel is where your files/data,

component toolset and task executions are displayed

By default, the resource panel contains the Userspace Tree, Components Tree and the Task Manager

To view/hide the Resources panel select View > Resources OR press Ctrl +2


Userspace Holds the listing of userspace files

Provides access to your workgroup's shared space

Where your data, projects, bookmarks, web links and reports are stored

This is similar to, but not the same as the file space that you typically access on your local machine

If the Userspace panel is not visible, use the View button > Userspace to show the Userspace Tree


Common Userspace Icons

User’s Private Folder

Workflow Project

Database Bookmark

Shared Folder

Data Table

Folder


Components Tree Holds the listing of all available

component nodes (building blocks) which can be used to construct workflows

The component nodes are sorted according to their functional group

Nodes and folders will vary according to installed plug-ins

Can also be accessed by right clicking in the project space

If the Components panel is not visible, use the View button > Components to show the Components Tree


Task Manager Provides you with

the means to control and track the execution of your tasks on the InforSense server Right click on an hourglass icon to halt a running task If the Tasks panel is not visible, use the View button > Tasks to show the Task Manager

Successfully completed

task

Successfully completed

task

Halted TaskHalted Task

Task in ProcessTask in Process


Work/Project Panel

Where you visually compose, edit and execute data analysis tasks, represented as workflows

Acts as a launch pad for invoking the execution of data-analysis procedures and the different data and model browsers/visualizers


Node Editor

Provides different tabbed panes of information about a selected node in the current Project panel depending on the type of node that is selected

Parameters, Input, Output, Notes, History, Interaction, Cache, User Customization

Toggle between these tabs by clicking on the tab buttons at the top of the properties editor panel

Use the View Button to restore any tabs that have been previously closed


Parameters Tab of the Node Editor

Most important tab in the Node Editor

Inspects, sets, edits and stores the values of the parameters for the different data tables and component nodes in a Project panel


Node Editor (Input/Output Tabs)

Provide information about the input and output metadata of a node


Node Editor (Cache Tab)

Once a node has been executed, the results are cached so that they don’t need to be re-executed, unless parameters have been changed

Clicking on the Build cache button executes the workflow up to the selected node


Node Editor (History and Notes)

Provides the history of the changes made to the parameters of a node

Allows the workflow builder to insert any comments or notes for future reference


Node Editor (Interaction)

Allows the workflow builder to require a user to enter a parameter value when the workflow is being executed

The workflow will pause when it gets to that point until the user enters the value


Navigator View

The Navigator panel provides an overview of the entire active project and can also be used for quick navigation of the project

If the Navigator Panel is not visible, select View > Navigator or press Ctrl+4


Customizing the Panels in the KDE Client

Resize the panels by dragging the double headed arrows between panels

Hide/Expand panels by clicking on the triangle icons

Close unwanted tabs

Tabs can also be dragged to split panes


Agenda

KDE Introduction







Case Study Exercise


Importing Tables Basic units of data in the system

Stored on the server

Consist of named, typed columns

Main column data types supported are Integer, Double continuous numeric, String, Dates, Vector

Can be loaded from:

Clipboard

Text Files (csv)

Excel files

Database (discussed in later section)


Import Tables with the Table Import Wizard

The Table Import Wizard can be invoked either:

From the File menu

Through the Table node


Table Wizard from the File Menu

Launch the

wizard from:

File

Import

Table from

File or

Clipboard


Table Import Wizard: Step 1

1. Select appropriate Import Data Source (For this example, use Text File)

2. Click Next

1

2



1. Select the table file to be imported from the appropriate path

2. Select appropriate delimiters depending on the file type

3. Select appropriate File Format

4. Click Next

1

2 3

4



1. Edit the data if needed in the Data Column settings panel

2. Select an appropriate Action for missing item

3. Select an appropriate Action for mismatched type

4. Click Next

1

2 3

4



Click Next



The Wizard imports the specified table and opens a Save Table Window

Click Finish



Save the table to the userspace in a folder giving appropriate name

Click Save to save the imported table for further use



To view the table, drag the saved table onto the Project panel

Right click on the table >Tools >Table Editor


Importing Non-Table Files

Non-Tabular data can also be imported into the Userspace using File > Import > Other File

No Wizard – just identify the userspace location and name

Useful for xml, sd, mol files, etc.

These alternative formats typically need an extra node to convert the format to a table added to the workflow


Although many data tables come from files, another common source is databases

Your InforSense administrator should have already configured the appropriate database connections to the KDE Server

If you are allowed access, you will also need to set up a database bookmark in the KDE client in order to access your database tables

Once the database bookmark is created you will be able to drag and drop database tables directly into a workflow and connect them to nodes for processing

There is no limit to the number of db bookmarks you can have

Importing Tables from Databases


Create a Database BookmarkA database bookmark can be created in multiple ways: Use the File Menu > New > Database Bookmark

Drag and drop the Database Bookmark node from the Components Tree into the project

Right Click in the Project panel. Select Import-Export > Database Bookmark to insert the Database Bookmark node into the Project panel


Enter the appropriate settings in the Database Bookmark Manager (if starting from the File Menu) or the Node Editor (if using the Database Bookmark node)

Click Test Database Connection

Check if Database is contacted successfully

If connection is successful, hit the Save button

Connecting the Database Bookmark


You will be prompted to choose the location in the userspace to store the database

Once saved, refresh the userspace from Resources menu to view the saved database in the userspace

Open the database schema up to find tables that can be dragged and dropped into workflows

Database Bookmark in the Userspace

jean

Need to get better db bookmark screenshot


In Database vs. KDE Server Processing

KDE processes of database tables can be done either on the KDE server (using standard KDE nodes) or within the database server (using a special set of IOE or RBDMS nodes, if licensed)

Processing data within the database server is generally more efficient

For best results, avoid many transfers of data between the KDE server and the database server

For the purposes of generality, this course will focus on nodes used only on the KDE server


Reading and Writing to Databases

Data tables from databases can directly read the source data

As KDE executes nodes on the KDE server, temporary tables with modified data will be stored on the KDE server

These temporary tables are NOT written back to the database to modify the original data except for the following nodes:

Table Data Manipulation

SQL

Export to Database

Oracle Table DML

Oracle Table Materialization

jean

check if there are other nodes...


Agenda

KDE Introduction







Case Study Exercise


Open a New Project

Select File > New > Project

OR

Click icon to open a new project

A blank Project page will appear in the Project Panel


Drag and Drop Tables and Nodes

Use the cursor to highlight an item from the Components Tree or Userspace

Drag the item into the project panel


Connecting Nodes The direction of arrows connecting the nodes indicates the logical flow of data during analysis

To connect nodes:Use the cursor to highlight the output port of the node or table

Left mouse click to view a connector arrow

Drag the connector arrow to the input port of the next node (all nodes eligible to be connected will be highlighted green)

Left mouse click on the desired node to save the connection

If a node has input requirements that are not satisfied, KDE will not allow the connection to be made


To disconnect nodes, right click on the connecting arrow (which will be highlighted in green)

Click Disconnect

OR

Right click on the first node

Click Disconnect

Disconnecting Nodes


Connecting Nodes via the Contextual Menus

As an alternative to dropping several nodes into the workflow and connecting them afterwards, you can:

Right mouse click on a selected nodeChoose Connect from the Contextual MenuRight click again to view the folders/nodes in the Components TreeSelect the desired node to be inserted into the workflow. It will automatically be connected to the selected node


Once the node connections are made in the workflow, each node should be checked to verify that the parameter values are set as desired

Click on each node in the workflow to select it

For each node, check the parameters tab in the Node Editor and enter or edit any required values

If a parameter value is missing or can not be run, error messages will appear at the bottom of the window

Customizing Node Parameters


Executing a Workflow Once the workflow is

constructed, you need to execute it in order to make the desired transformations

You can execute the entire workflow automatically by executing the last node in the line or you can execute just part of a workflow by executing an intermediate

Right click on the node and select Execute OR select node and hit green arrow in toolbar

You will be prompted for a location in the Userspace to store the results


During Workflow Execution During execution the

node in progress shows an hourglass icon to indicate the progress of execution

Completed nodes may have a green circle with a checkmark or have ‘[Cached]’ underneath

Workflow will continue to execute on the server, even if the client logs out

The Task Manager in the Resources panel displays the current status of the workflow being executed


After Workflow Execution

The saved table from the workflow execution can be seen in the Userspace

KDE usually automatically adds an icon to the Project Panel with a dashed line


Workflow Execution Symbols

Red circle with exclamation = An invalid parameter is set; workflow will not run until a valid value is entered

Double yellow lines = Workflow pauses during execution for user to enter parameter value while the workflow is being run

Green circle with checkmark = Node or table successfully run

Yellow triangle with exclamation = Build cache before proceeding; workflow needs to be executed to this point since the metadata is required for the next connected node


Re-Executing Workflows For each node that is executed (either explicitly or as an

intermediate), a temporary table is generated and stored as a Cache

When a workflow is re-executed, these temporary cache tables will be used for faster performance

If parameters are changed and you need to recalculate the tables, be sure to clear the cache (either from the contextual menu or the Cache tab in the Node Editor


Visualization Tools KDE supports a number of

powerful visualization tools to interactively view data

The tools that are visible in the contextual menu depend on the data type and the products licensed

Selecting a visualization tool will automatically execute the workflow up to that node, but does NOT store the table in the Userspace

To run a tool:

Select the node and output port

Right click on the node >Tools > Visualization tool


Another Way to Access Visualization Tools

Alternatively we can select a visualization tool from the menu bar (eye icon)

Click Tools and select a visualization tool


Common Visualization Tools Table Editor: Part of KDE Core, views the data in an Excel-

like table Text Viewer: Part of KDE Core, views the data in free text

format Interactive Browser: Part of KDE Core, user can construct

interactive plots with the data Classification Studio: Part of KDE Analytics, user can

interactively build classification models with the data Clustering Studio: Part of KDE Analytics, user can

interactively build clustering models with the data Spotfire DecisionSite: Requires Spotfire license, user can

construct interactive plots with the data ChemStudio: Part of ChemSense, user can construct

interactive plots with the data and run services


Table Editor can View (but not Edit) the Data

Table Editor Tab is used to view values in a table

Cell contents cannot be modified

Can click on node icons along top to run common pre-processing operations interactively


Table Editor Calculates Statistical Summary Select a

Calculate Button to generate statistics for a corresponding column

Provides quick statistical summary

Useful for identifying missing values and data spread


Saving a Workflow to the UserSpace

Workflow projects can be shared with other users if they are saved to the userspace

To save a workflow, select File > Save or Save as…

This opens a Save Project window


Choose the Location and Project Name

1. Select an appropriate folder where the workflow is to be saved

2. Give an appropriate name to the workflow

3. Click Save

32

1


Managing the Userspace It is helpful to organize related

tables and models into a folder with the workflow itself

Right click on a folder and select New > Folder to add a folder

Folders, projects, tables, etc. can be dragged and dropped into folders in the userspace

Right click on an item and select Rename to change a project to a descriptive name

Folders, projects, tables, etc. can be sorted in the userspace


Opening a Previously Saved Project

Right click on the project icon in the userspace and select Open

OR

Double right mouse click on the project in the userspace to automatically launch

Workflow should now be viewable in the Project Panel on the right


Agenda

KDE Introduction







Case Study Exercise


Login to the Discovery Portal

Default account is demo/demo Hit the ‘Login’ button to view the portal

Ask your administrator for the web address - it is usually <ServerIPaddress>:port number/kweb


KDE Discovery Portal Home Page

Use the menu bars to access functionality

Admin menu only visible if you are logged in as an administrator


Can View Projects with the Userspace Menu

Displays projects and database bookmarks Can drill down to get full execution details Can organize (delete/rename/move/etc.)

userspace from here


Can View or Kill Tasks in the Task Manager

Displays current and finished tasks Can halt current tasks from the portal


Using Deployed Workflows

Many end users will only work with workflows through services on the portal

Click on the ‘Browse Services’ option under the Services menu to view the available services


View Available Deployed Workflows

Click on a link to load the appropriate service page

A service is a workflow that displays a set of parameters that can be modified and the results when the workflow is executed


An Example Portal Service

Parameters to Set

Run buttons to execute parts of the workflow


Viewing the Results of an Example Portal Service

Depending on the configuration of the service, a link is displayed when the execution is completed

Click on the link to view the results


Agenda

KDE Introduction







Case Study Exercise



This section will be broken into four main subsections:

Overview of Workflow Deployment

Using the Promotion Wizard to expose parameters, inputs, and outputs to end users

Using the Publishing Wizard to deploy a workflow

Using deployed workflows in the Portal


Results generated from this node output

Concept of Workflow Deployment

Workflow Deployment allows users to encapsulate and expose the functionality of a workflow by:

Specifying which parameters should be editable in the workflow

Specifying which nodes in the workflow should results be generated from

Results generated from this node output

Columns to delete editable

Use table subset editable


Workflow Deployment Panel

Workflows are deployed using the Workflow Deployment panel

If the Workflow Deployment panel is not displayed, use arrow keys to open window or use View button to toggle off Actions display


Example Workflow to Deploy

Decide what parameters, inputs, or outputs to expose to end users

For our example workflow, we will promote the following items:

1. Use Sample parameter from the input table node2. Table output from the Filter node3. Columns parameter from the Delete node4. Table output from the Delete node

Table output port

Columns parameter

Use sample parameter

Table output port


Promoting a Parameter: Step 1

1

2

1. Select the Table node (to publish the ‘Use Sample’ parameter)

2. Invoke the Promotion Wizard from the Workflow Deployment panel by clicking on the ‘Promote’ button OR right click on the node icon and select ‘Promote to Workflow’

2



1. Select ‘Parameter’ as the item to promote

2. Click the ‘Next’ button to continue

1

2



1. Select the ‘Use Sample’ parameter to promote


1

2



1. Enable the Value checkbox for the ‘Use Sample’ parameter


1

2



1. Enter an appropriate parameter name and description (these will be displayed in the Portal later, so it is a good idea to rename in that context)


1

2



1. Review the promotion settings

2. Click the ‘Finish’ button to exit

2

1


Promoted Parameter is in the Parameters Tab

Promoted parameter appears in the Parameters tab of the Workflow Deployment panel

Repeat the same procedure for the ‘Columns’ parameter in the Delete node


Promoting an Output: Step 1

1

2

1. Select the Filter node (to publish the Table output)

2. Invoke the Promotion Wizard from the Workflow Deployment panel as before



1. Select the ‘Output port’ as the item to promote


1

2



1. Select the ‘Passed’ output as the port to promote


1

2



Accept the Associated Parameters default settings Click the ‘Next’ button to continue



Enter an appropriate Port name and description (these will be displayed in the Portal later, so it is a good idea to rename in that context)

Click the ‘Next’ button to continue

12



Accept the Connection Type default settings Click the ‘Next’ button to continue



1. Review the promotion settings

2. Click the ‘Finish’ button to exit

1

2


Promoted Output is in the Outputs Tab

Promoted output port appears in Outputs tab of the Workflow Deployment Panel

The output port on the node icon is now highlighted yellow Repeat the same procedure to promote the output table for

the Delete node


How to Edit Promoted Items: Step 1

1. Select the promoted item to edit in the Workflow Deployment panel

2. Launch the editor by clicking on the ‘Edit’ button

1

2


How to Edit Promoted Items: Step 2

1. Edit the appropriate promotion details

2. Click on the ‘OK’ button to commit changes and exit

1

2


Adding a Deployed Workflow Description

It is possible to provide a description for the deployed workflow in the Description tab

This description will be displayed in the portal


Publishing a Workflow Externally: Step 1

By default, deployed workflows are only available for use within the KDE client

In order to make any deployed workflow available to any external consumers (such as the portal), it needs to be published

Click on the Publishing Wizard button to publish the workflow

Note: The Publishing Wizard button is only visible for users who belong to the is-deploy role



1. Select the checkbox to publish the workflow to external consumers (Deselecting this checkbox will remove a published workflow from the portal)

2. Click on the ‘Next’ button to continue

1

2



Define how the deployed service is laid out in the Portal

The simplest way define a layout is to use the default pre-defined layout template

The default template aligns the promoted parameters and outputs vertically in the page according to the promotion order. The results are loaded in a separate new window

Click the Finish button to exit

1

3

2


Set the exposed parameters and hit the Results buttons to execute the workflow

View the Service in the Portal


Run the Workflow from the Portal

Click on the link to view the output table


Results in the execution output window of the Default layout

Output Table Displayed in the Portal


Agenda

KDE Introduction







Case Study Exercise


ThanksInforsense China

上海市钦州路 100号210235

Telephone: +86 21 6436 3311FAX:+86 21 5406 5058URL: http://www.inforsense.cn

InforSense KDE User Training Introduction to InforSense KDE and Building Workflows.

Documents

Transcript of InforSense KDE User Training Introduction to InforSense KDE and Building Workflows.