Cloud Computing for Geophysics: Virtualization of Infrastructure
TCM: Virtualization of Resources / Infrastructure
If you can't read please download the document
description
Solutions for Quasi-opportunistic Supercomputers G. Mécheneau Platform Computing [email protected]. TCM: Virtualization of Resources / Infrastructure. Takeaway. Focus of the project on Infrastructure for net-aware apps Real use cases (9) Virtualization of grid topology - PowerPoint PPT Presentation
Transcript of TCM: Virtualization of Resources / Infrastructure
PowerPoint Presentation
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
TCM: Virtualization of Resources / Infrastructure
Solutions for Quasi-opportunistic Supercomputers
G. McheneauPlatform [email protected]
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Takeaway
Focus of the project onInfrastructure for net-aware appsReal use
cases (9)Virtualization of grid topologyIs critical for complex
applicationsIs possibleExisting, deployable technologies
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Virtualization ?
The internet is a fantastic place
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Virtualization ???
A definition as presented by the science guy in:
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Virtualization !
Through Wikipedia :
Virtualization is the process by which"the supercomputer analyzes
your molecular structure through the scanners and breaks down your
atoms before digitalizing them and recreating a digital incarnation
in the virtual world."
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
No really, virtualization.
and breaks down your atoms
analyzes your molecular structure through the scanners
"the supercomputer
before digitalizing them
and recreating a digital incarnation in the virtual
world."
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
This work was supported by the EC grant FP6-2005-IST-5 033883
for the QosCosGrid project
QosCosGrid Vision:Grid virtual supercomputer
*
*
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
This work was supported by the EC grant FP6-2005-IST-5 033883
for the QosCosGrid project
*
*
QosCosGrid: 11 partners from 9 countries
AITIA InternationalInc., Hungary
Platform Computing Sarl, France
Poznan Supercomputing and Networking Centre (PSNC), Poland
Israel Institute of Technology (TECH)Israel
Universitat Pompeu Fabra (PFU) Spain
Cranfield University (CU),United Kingdom
Collegium Budapest (ColBud) Hungary
University of Amsterdam (UvA), Netherlands
University of Ulster (UU), United Kingdom
INRIA, France
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
This work was supported by the EC grant FP6-2005-IST-5 033883
for the QosCosGrid project
*
*
Systems composed of interacting componentsSimple entities yield
complicated dynamicsNonlinearity, self-organization (pattern
development)The whole is more than the sum of its partsRecursive
effects from interactions; path dependence; dynamically emergent
propertiesTypically not amenable to analytic solutionsSize and
computational complexity, explosionNon-existence of solution:
infinitely long lived transients, non-equilibrium cascades,
sensitive dependencies, etc.
Complex Systems, Definitions
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
What for ?(Or: Our target use cases)
Complex SystemsSystems composed of interacting components"The whole
is more than the sum of its parts"Typically not amenable to
analytic solutions
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
This work was supported by the EC grant FP6-2005-IST-5 033883
for the QosCosGrid project
Complex Systems Simulations (Examples)
N-Particle SystemsProtein InteractionsMetabolic PathwaysFinancial
MarketsMarket ResearchSupply Chain OptimizationEcological and
Population DynamicsStellar Systems
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Virtualization in practice
Description languageFor the jobsFor the grid
Programming environnement
Resource management : fabric
MetaschedulerAble to understand workflows of simpler
entities
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
This work was supported by the EC grant FP6-2005-IST-5 033883
for the QosCosGrid project
Architecture Overview
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
This work was supported by the EC grant FP6-2005-IST-5 033883
for the QosCosGrid project
QCG Parallel Toolkit
Implementation
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
This work was supported by the EC grant FP6-2005-IST-5 033883
for the QosCosGrid project
Testbed
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
This work was supported by the EC grant FP6-2005-IST-5 033883
for the QosCosGrid project
On-demand resource acquisition and formation of application
specific Grids: QosCosGridQosCosGrid (= Quasi Opportunistic
Supercomputing for Complex Systems on the Grid) Complex Systems
applications use the QosCosGrid-Toolbox to parallelize their
workload in order to use distributed resources. Towards the
QosCosGrid-Broker applications express requirements and behavior by
xml job profileThe Broker acquires resources on-demand and form an
application specific Grid, creates RTG = resource topology graphRTG
is used to map the application to the resources, placing MPI
communicators at the right place
QosCosGrid workflow
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
This work was supported by the EC grant FP6-2005-IST-5 033883
for the QosCosGrid project
NExTml: QCG Job profile
The QCG Job Profile document is inspired by an existing XML-based
job description language supported by one of the main components of
the QCG middleware called GRMS. End users can describe topology and
resource requirements, in particular:required aggregations and
hierarchies of resources (computing nodes, clusters, sub-clusters,
storage elements etc.),required resource properties (operating
system, memory, number of CPUs, speed of the CPU on a
resource),required network and connection properties (bandwidth,
latency and capacity),required applications and licenses available
at destination computing resources.
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
This work was supported by the EC grant FP6-2005-IST-5 033883
for the QosCosGrid project
NExTml :QCG Resource Description Model
RTG (Resource Topology Graph) A common XML resource description
language Provide description of:Resources, tasks, processesTopology
Communication properties Serves as a bridge between the various
system componentsUsed to describe, publish, evaluate, reserve and
monitor heterogeneous resources across the QosCos Grid
Supplementary Java implementation:Functional behavior and
logicXML to Java objects marshaling/un-marshalingSpecialized types
of RTG objects, according to the middleware requirements (i.e.
Resource advertisement, Meta-scheduling, SLAs, Monitoring, and User
requirements.)
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Example job on a QosCosGrid
Write your descriptionSubmit it to GRMSGRMS queries existing Grid
Topologyallocates tasks of the workflow on the grid
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Example job on a QosCosGrid
Usecase 8: AITIAPredator-prey ecologyCellular Automaton
(CA)Partitioning Divide CA Adapt to available number of nodesMaster
/ slave divisionUsing QCGProactive
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
NExTml descriptor - Topology
8 ... ... 1
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Example QosCos Grid
Cluster
Title
Cluster
Title
Cluster A
WAN
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Job scheduling
Cluster
Title
Cluster
Title
Cluster A
WAN
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Job scheduling
Cluster
Title
Cluster
Title
Cluster A
WAN
AD Information service
AD Information service
3
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Job scheduling
Cluster
Title
Cluster
Title
Cluster A
WAN
AD Information service
AD Information service
3
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Job scheduling
1
1
Cluster
Title
Cluster
Title
Cluster A
WAN
AD Information service
AD Information service
3
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
PlugnPlay deployment
Our target: "as simple as firing up a VM"Existing resource manager
in place does not changeLSF, PBS, MauiFull control of resources is
maintainedNo additional deployment on the grid nodes, no kernel
changes, no specific linux. No nothing.Just deploy a QCG Head &
allocate a budget
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Oui Nide Iou
Further adoption of NExTml
Further deployments
Porting to other schedulers, dev env, etc.
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Performance
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Performance
This work was supported by the EC grant FP6-2005-IST-5 033883 for the QosCosGrid project
Performance
*
*
Job Profile = User->App requirements->Broker the job profile
is used by the user to declare application requirements
RTG = Broker->Resource properties-> map application to resourcesthe broker describes by the resource topology graph the acquired resources
the application is implicitely 'mapped' on to the graph.the
graph declared by the job profile may map the different
communicators in an MPI jobThe actual allocation of nodes to the
job will try and match this topology, by placing nodes belonging to
given communicators at the appropriate place
*
Job Profile = User->App requirements->Broker the job profile
is used by the user to declare application requirements
RTG = Broker->Resource properties-> map application to resourcesthe broker describes by the resource topology graph the acquired resources
the application is implicitely 'mapped' on to the graph.the
graph declared by the job profile may map the different
communicators in an MPI jobThe actual allocation of nodes to the
job will try and match this topology, by placing nodes belonging to
given communicators at the appropriate place
*
Job Profile = User->App requirements->Broker the job profile
is used by the user to declare application requirements
RTG = Broker->Resource properties-> map application to resourcesthe broker describes by the resource topology graph the acquired resources
the application is implicitely 'mapped' on to the graph.the
graph declared by the job profile may map the different
communicators in an MPI jobThe actual allocation of nodes to the
job will try and match this topology, by placing nodes belonging to
given communicators at the appropriate place
*
*
*
*
Job Profile = User->App requirements->Broker the job profile
is used by the user to declare application requirements
RTG = Broker->Resource properties-> map application to resourcesthe broker describes by the resource topology graph the acquired resources
the application is implicitely 'mapped' on to the graph.the
graph declared by the job profile may map the different
communicators in an MPI jobThe actual allocation of nodes to the
job will try and match this topology, by placing nodes belonging to
given communicators at the appropriate place
*
*
*
*
*
*
*
