1 Logistical Computing and Internetworking: Middleware for the Use of Storage in Communication Micah...
-
Upload
josephine-lindsey -
Category
Documents
-
view
212 -
download
0
Transcript of 1 Logistical Computing and Internetworking: Middleware for the Use of Storage in Communication Micah...
1
Logistical Computing and Internetworking: Middleware for the Use
of Storage in Communication
Micah BeckJack Dongarra Terry MooreJames Plank
University of Tennessee
Fran BermanHenri Casanova
University of California, San Diego
Rich Wolski
University of California, Santa Barbara
3rd International Workshop on Active Middleware Services
2
LoCI Projects
• Internet Backplane Protocol (Beck, Plank)
• Network Weather Service (Wolski)
• NetSolve (Dongarra)
• Application Level Scheduling (Berman)
LoCI Funded by National Science Foundation Next Generation Software Program
3
Internet Backplane Protocol (IBP)
• primitive middleware that supports a layer of network storage
• implemented as a system of buffers exposed for direct scheduling,
• can be used by advanced applications to leverage state management for high-performance.
4
Network Weather Service (NWS)
• Monitors and extrapolates network metrics– network bandwidth and latency– storage availability– CPU load
• Prediction is weak reservation– all reservations will sometimes be broken– effective for highly aggregated resources
5
NetSolve (NetSolve)
• Provides a programming environment that facilitates the analysis of program dependences to understand an application’s inherent communication requirements.
• A major component of LoCI research is identify and provide opportunities for extracting scheduling information from applications.
6
Application Level Scheduling (AppLeS)
• Enables the derivation of an efficient schedule that matches communication requirements.
• Mapping the computation, network and storage resources of the application to the Grid resources subject to current and predicted resource conditions, is a difficult problem.
• AppLeS is the leading instance of a range of approaches we are exploring under LoCI.
7
An Analogy with Pipelined Processor Architecture
• The fundamental elements of modern processor architecture are:– Buses and functional units which move and
transform data, and – Memory and cache, registers and pipeline
buffers that store data.
• RISC architecture exposes resources to scheduling by the compiler
8
Network Computing has Analogous Components
• In our model of logistical network computing, the fundamental elements are– Predictable networking and computation which
move and transform data, and– Storage that is accessible from the network.
• Logistical Computing exposes resources to external schedulers (including applications)
9
Logistical Networking: Exposed Storage Management
• Storage resources available for direct access at network intermediate nodes.
• Allocation and scheduling of storage resources are exposed to the network.
• Some implications– storage resources are shared among operations– applications, intermediate nodes can schedule
10
IBP Software Structure
• IBP Depots (servers) are daemons that serve local storage to IBP clients.
• IBP Clients link an IBP client library with a well-defined API.
• Clients talk to depots using TCP/IP.
• Design is for high-performance/scalability.
11
Logistical Computation Mechanisms
• The Network Weather Service: Monitoring Resources for Logistical Scheduling
• Logistical Scheduling and the AppLeS Project
• Coscheduling of Storage and Computation in NetSolve
12
NetSolve - The Big Picture
Reply Choice
Computational ResourcesClusters
MPP
Workstations
MPI, PVM,Condor...
RequestAgent
Scheduler
Database
Client - RPC like
MatlabMathematicaC, Fortran Java, Perl Java GUI
13
State Management in NetSolve
• The Problem: NetSolve calls are functional
• Excessive data transfers
For example:X = F(A, B);Y = G(X, B);
ClientA,B
F
ClientX,B
G
Client
X
Y
Server 1
Server 2
14
ClientA
F
G
Client
Y
Server 1
Server 2
Client A,B
F
A,B
G
Client
Y
Server 1
Server 2
X
X,B
Y
Caching
X
B
B
B
Dependence Flow
IBP
Cache
15
An Experiment Using NetSolve
• NetSolve Client at UC San Diego
• Computational Servers at UT Knoxville
• MA28 solver library used to solve systems of equations from the Harwell-Boeing collection of the Matrix Market repository
• Uncached to client-directed caching
16
Preliminary Results
• Unenhanced NetSolve vs. NetSolve w/IBP caching
16.1 KB 2.68 MB
17
LoCI Software Integration
• IBP Depot (server) available for Unix/Linux and Win32
• IBP Client Library also available for Java
• NetSolve 1.4 (just released) supports IBP caching
• Network Weather Service uses IBP internally for monitor state management
18
Conclusions
• Logistical Computing defines a comprehensive exposed approach to Grid computing
• Processing, network, and storage resources are explicitly scheduled for performance
• Storage resources sharing enables improvements over stateless computation based solely on end-to-end communication