Tool Dæmon Protocol© 2002 Barton P. Miller March 4, 2001

The Tool Dæmon Protocol:Using Monitoring Tools on

Remote Applications

Barton P. Millerbart@cs.wisc.edu

Computer Sciences DepartmentUniversity of Wisconsin

Madison, Wisconsin 53705USA

– 2 – Tool Daemon Protocol© 2002 Barton P. Miller

The Challenge

Consider remote process management environments:

CondorMPI

• Portable MPI (such as MPICH)• Vendor provided MPI (such as IBM, Compaq,

Sun)Globus

Each of these environments needs to monitor and control the state of its application processes.

– 3 – Tool Daemon Protocol© 2002 Barton P. Miller

Typical Process Manager

Process manger:• Starts the

remote job• Monitors its

status• Controls the job• Sets up file I/O• Sets up standard

I/O

Remote ProcessManager

monitor/control

ApplicationProcess

ApplicationProcess

Remote Host

– 4 – Tool Daemon Protocol© 2002 Barton P. Miller

Typical Process Manager

Remote ProcessManager

monitor/control

ApplicationProcess

ApplicationProcess

Remote Host

Tool DæmonProcess

?

?

The run-time tool?• Also may want to

start process (or attach to it)

• Also needs to monitors its status

• Also may want to control the job

• Needs to communicate with its front-end.

– 5 – Tool Daemon Protocol© 2002 Barton P. Miller

Typical Process Manager

Remote ProcessManager

monitor/control

ApplicationProcess

ApplicationProcess

Remote Host

?

?

Tool DæmonProcess

So, who wins?

– 6 – Tool Daemon Protocol© 2002 Barton P. Miller

Typical Process Manager

Remote ProcessManager

monitor/control

ApplicationProcess

ApplicationProcess

Remote HostLocal Host

?

?

ToolFront-EndProcess

Tool DæmonProcess

– 7 – Tool Daemon Protocol© 2002 Barton P. Miller

Current State of AffairsEach process manager starts and controls

processes in its own way.• E.g., even within MPI: IBM POE MPI, SGI Origin MPI

and MPICH all work differently. MPI has no standard process control!

Specialized cases of a specific tool working with a specific environment• e.g., TotalView debugger working with MPICH.

The result is an m n combination of m process managers and n tools.

Bottom line: need a standard interface for process managers and tools to coexist: the Tool Dæmon Protocol (TDP).

– 8 – Tool Daemon Protocol© 2002 Barton P. Miller

The Basic TDP Steps1. Create, but don’t start, new application process.2. If necessary, create tool daemon process.3. Pass basic information to tool daemon: e.g.,

1. Application PID.2. Front-end host/port number.3. Standard I/O host/port number.

4. Tool daemon processes application:1. For a debugger, read symbols2. For Paradyn/dyninst, parse the executable.

5. Start the application process6. Respond to changes in the application state.7. Respond to changes in the tool daemon’s state.

– 9 – Tool Daemon Protocol© 2002 Barton P. Miller

Step 1a: Request Process Creation

Remote HostLocal Host

ToolFront-EndProcess

Remote ProcessManager

Create process

– 10 – Tool Daemon Protocol© 2002 Barton P. Miller

Step 1b: Create Application Process

ApplicationProcess

Remote HostLocal Host

ToolFront-EndProcess

Remote ProcessManager

– 11 – Tool Daemon Protocol© 2002 Barton P. Miller

Step 2: Create Tool Dæmon

ApplicationProcess

Remote HostLocal Host

Tool DæmonProcess

Remote ProcessManager

ToolFront-EndProcess

– 12 – Tool Daemon Protocol© 2002 Barton P. Miller

Step 3: Pass Info to Tool Dæmon

ApplicationProcess

Remote HostLocal Host

Remote ProcessManager

ToolFront-EndProcess

PID, host/port pairsTool Dæmon

Process

– 13 – Tool Daemon Protocol© 2002 Barton P. Miller

Step 4: Process the Application

Remote HostLocal Host

Remote ProcessManager

ToolFront-EndProcess

Tool DæmonProcess

ApplicationProcess

– 14 – Tool Daemon Protocol© 2002 Barton P. Miller

Step 5: Start the Application

Remote HostLocal Host

Remote ProcessManager

ToolFront-EndProcess

Tool DæmonProcess

ApplicationProcess

– 15 – Tool Daemon Protocol© 2002 Barton P. Miller

Interesting Challenges

1. Return codes and state changes?

2. Who control standard input and output?

3. Private networks and firewalls:

1. Tool dæmon communicating to front-end.

2. Application process sending standard I/O.

– 16 – Tool Daemon Protocol© 2002 Barton P. Miller

Challenge: Firewalls and Private Nets

Remote HostLocal Host

Remote ProcessManager

ToolFront-EndProcess

Tool DæmonProcess

ApplicationProcess

Firewall

X

– 17 – Tool Daemon Protocol© 2002 Barton P. Miller

Challenge: Firewalls and Private Nets

Remote HostLocal Host

Remote ProcessManager

ToolFront-EndProcess

Tool DæmonProcess

ApplicationProcess

Firewall

CommProxy

– 18 – Tool Daemon Protocol© 2002 Barton P. Miller

Challenge: Firewalls and Private Nets

When tool daemon is started, pass in the host/port number of its front-end process.

If there is a communication proxy, then:• Tool daemon will receive host/port of the

proxy, so daemon connects to proxy.• Proxy will connect to the tool front-end,

mapping the host/port (similar to NAT).Application connecting to console for

standard I/O works the same way.

– 19 – Tool Daemon Protocol© 2002 Barton P. Miller

The Condor/Paradyn Scenario

Condor Starter

monitor/control

ApplicationProcess

ApplicationProcess

Remote HostLocal Host

ParadynFront-End

Paradyn Dæmon

– 20 – Tool Daemon Protocol© 2002 Barton P. Miller

The Path ForwardProduce a prototype implementation to

expose technical challenges:• Paradyn running under Condor• Ana Cortes and Miquel Senar (UAB/UW)• A presentation tomorrow about the first steps.

Goal is to produce a standard set of libraries for process managers and tool daemons.

Involve a wider community in this standards effort• Initially: ANL (Gropp and Lusk), Etnus (Cownie

and Delsignore), Globus (Kesselman)

– 21 – Tool Daemon Protocol© 2002 Barton P. Miller

The Path Forward

This is just the beginning of an important effort

Your input and participation is welcome and encouraged.

This project is supported by the hard work and advice of the Condor and Paradyn teams!