Dept. of Computer Science, NC State University http://www.cbl.ncsu.edu/OpenProjects/OmniFlow/

**H. Lavana is now with Cisco Systems, Inc.

A Universal Client for Taskflow-Oriented Programming with

Distributed Components: Concepts

Franc Brglez Hemang Lavana**

Outline

• DA Background and Motivation

• Taskflow Architecture

• Taskflow Programming and GUI Highlights

• Summary and Conclusions

DA Background and Motivation ….

• Informal problem description

• Matching S/W components to task instances

• Informal taskflow construction for

-- concurrent and distributed task execution

-- serial and distributed task execution

Informal Problem Description

Task A1: std. cell placement of netlist N(a) (@host a1)

Task A2: std. cell placement of netlist N(a) (@host a2)

Task B1: block placement of netlist N(b) (@host b1)

Task C1: routing of placed N(a) & N(b) (@host c1)

Task D1: N(a) & N(b) post-layout simulation (@host d1)

Notes: • tasks may be hierarchical, owned by distributed team • hosts may be under different file systems !!

(3) when C1 completes, -- invoke and execute D1

What are the key challenges in this example?

(1) invoke and execute {A1, A2, B1} concurrently

(2) when first 2-of-3 from {A1, A2, B1} complete -- abort uncompleted task from {A1, A2, B1} -- invoke and execute C1

Matching S/W Components to Task Ins.

Repeat Task A1

(@a1)

Repeat Task B1

(@b1)

Repeat Task A2

(@a2)

Sel 2-of-3 Task C1

(@c1)

Do Task D1

(@d1)

Repeat Task A1

(@a1)

Repeat Task B1

(@b1)

Repeat Task A2

(@a2)

(1) invoke and execute {A1, A2, B1} concurrently ….

Beg

(0) invoke

taskflow

Concurrent Taskflow Composition

(2) when first 2-of-3 from {A1, A2, B1} complete ….

Sel 2-of-3 Task C1

(@c1)

-- abort uncompleted task A1

-- invoke and execute C1

Taskflow Aborts and Synchronization

Repeat Task A1

(@a1)

Repeat Task B1

(@b1)

Repeat Task A2

(@a2)Beg

(3) when C1 completes ….

Do Task D1

(@d1)

-- invoke and execute D1

EndEnd

(4) terminate taskflow

Repeat Task A1

(@a1)

Repeat Task B1

(@b1)

Repeat Task A2

(@a2)Beg

Sel 2-of-3 Task C1

(@c1)

Serial Taskflow Composition

Note: with Beg/End tasks, a taskflow is a polar graph ...

Related Work

• University-based research, e.g. Reuben (DAC’97), Weld (DAC’98), JavaCADD (TR’98), CollabTop (DAC’99), OpenDesign (ICCD’00), OmniFlow XML/TclTk Implementation (TclTk’01)

• Structured Workflow Modeling and Middleware, e.g. http://www.icis.qut.edu.au/~arthur http://www-4.ibm.com/software/ts/mqseries/

• Technologies such as XML, OMG, JavaSOFT, COM ….

• Collaboration technologies, e.g. http://www.cpsc.ucalgary.ca/grouplab/groupkit/

Taskflow Architecture & Programing ...

• Problem Abstraction

• Blackbox (Whitebox) as Component Instances

• Taskflow as Intersection of TaskGraph, DataGraph

• Task Encapsulation Architecture

• cdtML Taskflow Schema in XML

Problem Abstraction

Given: L blackbox components (executable programs) and data; M whitebox components (compositions of black/whitebox components) and data;

K hosts of L+M black/whiteboxes;

N owners of L+M black/whiteboxes.

(1) compose a new whitebox component;(2) transfer data within the new whitebox component;(3) schedule concurrent/serial execution of all component instances contained in the new whitebox component.

Three problems:

WhiteBox as a TaskFlow of Components

- tasks can be sequenced serially or concurrently

- the choice of sequence can be decided by data - task may be repeated a number of times

- only one entry and one exit point

- enabling/disabling of control edges to control the scope of execution- enabling/disabling taskflow for execution with local or global flow data

Platform-independent: accessible from a web browser

Programmable:

Interactive, user-configurable during runtime:

Collaborative: asynchronous/synchronous mode

Task A1 (@a1)

temporary

Data types: persistent

Control ports: task invocation task abort task status

Data ports: input

output

Black/WhiteBox Ports and Data Types

inout

outin

Task A1 (@a1)

Task B1 (@b1)

BlackBox/WhiteBox Connectors (Edges)

transfer data edge (data-task, task-data or data-data)

singleInvocation control edge (task-task, opened/closed in GUI)

repeatInvocation control edge (task-task, opened/closed in GUI)

abortInvocation control edge (task-task, opened/closed in GUI)

Taskflow: Intersection of Two Graphs

End Join Task

Inst.3

• Each task instance encapsulates FIVE primitives

• User can re-configure (on/off) all control edges

Task Inst.1

Beg Fork

(1) ControlGraph

Task Inst.2

AND (2) DataGraph

Task Instance Architecture

Task Instance Layer

Encapsulation Layer

BBC/ WBC

FSMD Control Fork

… to other tasks

Control Join

… from other tasks

Data Mux

local/flow data

Task Encapsulation Layer (Outline)

Finite-State-Machine as per FSM transition table:

completion status

BBC/ WBC

invoc/abort signal

••••Datapath

(as per datapath table)

••••

Din Dout

{ Waiting, Enabled, Xecuting, Completed Done, Aborted, TimedOut, Skipped }

cdtML Taskflow Schema (using XML)

BeginForkTaskInst1 …...EndJoinDataGraph

TaskInstListTaskGraph

I/O PortList …...

I/O PortList …...

TaskInstance

SingleTaskBody

SingleTaskDefn

Encapsulated SingleTask

BeginForkBlackBox Cp.EndJoin

Encapsulated MultiTask

MultiTaskBody

MultiTaskDefn

TaskInst1

TaskInstListTaskGraphBeginFork

EndJoin …...

MainTask

ControlJoinDataMuxEncaps.TaskRepeatCond.ControlFork

Taskflow GUI and Scheduling Engine

cdtML Schema

cdtML Taskflows

Taskflow Scheduler

Tcl-xml Parser

Validate Taskflow Specs

Display Taskflow Tree, Graph

Begin

Enable Task

Execute BBC or expand WBC

Foreach child task

No more tasks End

Evaluate ControlJoin

FalseNoOp

Evaluate ControlFork

FalseNoOp

Repeat until

User

A Summary of Simplified Definitions

A task or a taskflow is an instance of

an encapsulated whitebox component

an encapsulated blackbox component

or

A taskflow schedule is a dynamically generated sequence

of concurrent and serial task invocations/aborts

(any legacy program accesible via TCP protocol using telnet-, ssh-, http-, socket-based clients)

(a directed polar graph of blackbox and whitebox components)

Taskflow Programming and GUI ...

• early DA projects reported in ICCD'00, demos at DAC’99, DAC'00 • a number of projects reported in PhD Thesis'00 (largest taskflow has 9150 task instances, longest path of 1600 task instances) • demos at DAC'01

Major GUI Views and evaluations (this talk):

• selector, tree, graph, status, info • graph view for interactive programming • scalability of taskflow scheduler

- (taskFlowInstance1 - InputList

- OutputList

- TaskList

Traditional display of hierarchy …. (follows cdtML schema)

• mIn1 = “fileIn1.t”• mIn2 = “fileIn2.t”

• mOut1 = “fileOut1.r”

|x||o|x|

|x|

|o|x|

• (Begin) - (TaskA)

+ InputList+ OutputList+ TaskList

- (TaskB) + InputList+ OutputList

+ (TaskC) • (End)

open graphView

} single task

} multi task

invoke task

Taskflow Tree View (Dyn. Generated)

A graph view, shown next, depicts additional user-configurable dependencies ….

INVOKE: DATA:sim.(noData)

exec.(locData)

exec.(flowData)

view

edit

Selector View

Taskflow Graph View (Dyn. Generated)

(End)(TaskC)

(TaskB)

blackbox

(Begin)

user- controlled

edges

whitebox(TaskA)

Click on any task:- to invoke- to abort

Click to view/edit

data

8-state color code

Interactive Taskflow Programming

(End)(Begin)

(TaskB)

(TaskC)

(TaskA)

(TaskC) (End)

Runtime switch from concurrent

(Begin)

to serial execution

(Begin)

user disconnects control edges

(TaskA)

(TaskB)

(TaskA)

(TaskC)

Can do similarly for abort and repeatInvocation edges

Invoke1

Invoke2

Taskflow Instance (cdtXML Start-Up)

Start-up Tree View Start-up

Graph View

Selector View

Status View

user-probing of DataGraph

INFO View

Taskflow Inst. (re-configured in GUI)

disabled invocationEdge

disabled repeatInvEdge

Enabled concurrent execution of tasks {(InitB), (InitC)}, then {(B), (C)}

“green” state shows task completion

Scalability of the OmniFlow Scheduler Task Longest Overhead Instances Path [seconds/task] -------- ------ ------------

9150 1600 0.381

300 100 0.396 300 200 0.412 2400 100 0.408

high concurrency taskflow

low concurrency taskflow

2400 1600 0.384

… for a SUN workstation under unix/solaris. Comparable performance has been observed on workstations under linux/macOS/windowsNT

Summary and Conclusions

• Well-defined encapsulation of b/w box component instances (via FSMD, xFork, xJoin, DataMux) and interfaces (inv/repeat/abort edges) supports

For more info about OmniFlow client, thesis, and paper reprints/preprints, see

http://www.cbl.ncsu.edu/OpenProjects/OmniFlow/

-- an effective XML/TclTk implementation of the universal OmniFlow client (companion paper).

-- hiearachical taskflow construction-- serial/concurrent task scheduling-- highly configurable GUI, dynamic taskflow execution