Grid - korisnicki pristup i razvoj aplikacija

www.see-grid-sci.eu

SEE-GRID-SCI

The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no. 211338

Grid - korisnicki pristup i razvoj aplikacija

Branko MarovicRCUB - UoB

Meteo VO Training, Belgrade24. June 2008

Application Gridification 2/26

SEE-GRID-2 Applications Adaptation & Support

Application lifecycle defined in “Gridification Guidelines” document.SG Gridification Guide (wiki pages) structured and contributed by developers.Policy: Applications initially supported at the country level.Application Support Group (ASG) created for gridification support.Application Review Committee(ARC) created to assess existing and new candidate applications.Application selection criteria defined by ARC.MoU Between Applications and SEEGRID-2.


SEE-GRID-2 Strategy for New Applications

There is “Gridification Guideline” document to provide initial help to developers. (application life-cycle, policy and methodology to run and support an application on Grid)There is “SG Gridification Guide” on SEE-GRID-2 Wiki to provide comprehensive help to developers (porting an existing application to the Grid)ARC (Application Review Committee) recommendations push applications towards collaborations with other applications and projects.Applications are required to provide update on their established user community and collaborations within their periodic Application Progress Report.


SEE-GRID2 Applications


Support the deployed applications

Application support group (ASG) created to provide fast and continuous gridification support to developers. Experienced developers & admins ASG team members assigned for each application. National level application support SEE-GRID - global level application support

SG Gridification Wiki online.SEE-GRID helpdesk available and used for problems. Useful Helpdesk Tickets for Developers. SEE-GRID Helpdesk “Applications Support” knowledge base. GOODs responsible to dump useful tickets of the week.

Work in close collaboration with WP5 (training) and WP3 (software requirements, maintenance of performance)


Application Lifecycle


SEE-GRID-2Gridification Guides

An initial guide for the developer including the application life-cycle, VO and site policies, gridification methodology: Gridification Guideline http://www.see-grid.eu/content/modules/

downloads/SEEGRID2-WP4-TR-008-Gridification_Guideline-n-2006-11-22.doc

Detailed wiki pages with illustrations for gridification issues: SG Gridification Guide http://wiki.egee-see.org/index.php/

SG_Gridification_Guide


SEE-GRIDGridification Guide

Relevant topics for application developers identified trough online questionnaire systemSome investigation areas identified as well – candidates for future GG topicsGridification guide will provide information on these topicsWiki as collaboration medium


SEE-GRIDGridification Guide

1 Job Management 1.1 Running Jobs (T1) 1.2 Software Installation Management (T3)

2 File Management and Access 2.1 Data Management (T2) 2.2 Data Access (T4) 2.3 Logical Files (T6) 2.4 Advanced Replica Management (T2, T6 related investigation area)

3 Inter-Job Communication 3.1 Workflows (T7) 3.2 MPI (T9) 3.3 Interactive & Pilot Jobs (T1 related investigation area) 3.4 Java networking

4 Information Services 4.1 Application-Level Event Logging & Performance (T5)

4.1.1 Applications and Systems monitoring tools 4.1.2 Benchmarking

4.2 Metadata Catalogues 5 Development Environments and Portals

5.1 Web Authentication and Authorization (T8) 5.2 P-GRADE Portal (T1-T9 related investigation area) 5.3 P-GRADE (T1-T9 related investigation area)


VO Resource Allocation Policies

Seegrid VO is used for all applications: Utilization of groups and roles:

/seegrid/COUNTRY_CODE/SHORT_APP_NAME/Role=production/seegrid/COUNTRY_CODE/SHORT_APP_NAME/Role=software

For gridification testing phase, no role or group is used, all sites support all

applications.

For deployed applications, sites are encouraged to support as many

applications as possible.

MoU template between SEE-GRID-2 Project and the Application created.


SEE-GRID-SCI

Project administrative and technical management

Adapt Grid applications

Support the deployed

applications

NGI support and international collaboration

Dissemination and Training

JRA1NA4User communities support

NA1

NA2

NA3

Application extensions deployment and assessment

Implementation of application-specific

services

Assess applications Grid usage and take-up

Development of application-level services and operational tools

SA1eInfrastructure operations

Application requirements analysis

Design, implementation and assessment

of the application-specific

monitoring and control tools

Operational tools analysis

Resource Centre Service Level Agreement monitoring and enforcement

Network Resource Provision

Implementation of the advanced SEE-GRID infrastructure

Study the Grid applications


Meteorology VO

Members National Observatory of Athens (coordinator) South Environment and Weather Agency of Serbia Hydrometeorological Institute of Montenegro RBI of Zagreb University of Zagreb Federal Hydro-Meteorological Institute of Bosnia and

Herzegovina Republic Hydrometeorogical Institute, Banjaluka

Two sets of applications for initial deployment: Regional scale Multi-model, Multi-analysis ensemble

forecasting system Study of the interaction of airflow with complex terrain


Developer Resources

Grid environment is constantly evolving, but Useful features persist New are constantly being added Bugs are being fixed Gained knowledge remains relevant, must be updated Applications can be easily migrated to new/updated APIs

gLite User Guide https://edms.cern.ch/file/722398/1.2/gLite-3-UserGuide.pdf

WMS user guides https://edms.cern.ch/document/572489/1

SEE-GRID Gridification Guide http://wiki.egee-see.org/index.php/SG_Gridification_Guide

SEEGRID Wiki http://wiki.see-grid.eu/

gLite documentation http://glite.web.cern.ch/glite/documentation/


Grids: a foundation for e-Research

Enabling a whole-system approachCollaborative research / engineering / public service …

sensor nets

Shared data archives

computers

software

colleagues

instruments

Grid

Diagram derived fromIan Foster’s slide


Načini upotrebe grida

Komandna linija, JDL, input/output sandbox i/ili ručna manipulacija podacimaPrimena skriptova u generisanu poslova, obradi i manipulaciji podacimaKorišćenje postojećih Grid portalaPrimena grid API-ja i servisa iz koda koji se izvršava na griduPravljenje aplikativno specifičnih pristupnih tačaka-servisa


Complexities of grid applications

1. Simple jobs – submitted to WMS to run in batch mode2. Job invokes grid services

• To read & write files on SE• Monitoring• For outbound connectivity (interactive jobs)• To manage metadata• …

3. Complex jobs• An environment controls multiple jobs on users’ behalf

• High-level services• Portals with workflow • Software written for the VO (or by the user)• …


Invocation of applications

From the UI Command Line Interfaces / Scripts APIs Higher level tools

From desktop Windows applications Use Grids without awareness of them! But gLite not (yet) fully supporting Windows (more info: http://jessica.trigrid.it/grid2win/)

From portals For recurring tasks: “core grid services” as well as application layer Accessible from any browser Tailored to applications Different portal solutions, and wide range of capabilities. Second part

of this course: P-GRADE Portal


Characteristics of VOs

What is being shared? resources of storage and/or compute cycles software and/or data

Distinct groups of developers and of users? Some VOs have distinct groups of developers and users…

Biomedical applications used by clinicians,…. …. Some don’t

Physics application developers who share data but write own analyses

Effect: need to hide complexity from the 1st type of VOs expose functionality to 2nd type of Vos many security issues


Challenges

Research software is often Created for one user:

the developer Familiarity makes it

useable Short-term goals: Used

until papers are written and then discarded

Grid applications are often used by a VO Without support from

developer In new contexts and

workflows

Need expertise in:

• software engineering

• application domain

• grid computing

Grid application developers are In a research

environment Yet their s/w must

have: Stability Documentation Usability Extendability

i.e. Production quality


Consequences

Team work!Engaged in world-wide initiatives – reuse, don’t make your own! Cross disciplines for solutions.From research to production software: ~5 times the effort. “80% of the time for last 10% of the functionality & reliability”

Standardisation is key For re-use, for dynamic configuration of services,.. Both for middleware and domain specific

Need to follow a deliberate development process Waterfall? Rapid prototyping? Requirements engineering, design, implementation, validation,

deployment Engaged with the user community


Basic tasks while Porting applications to the Grid

1. Developing a non-grid application (or inheriting and updating an ancient one);

2. Go/no-Go decision about gridification• Is it suitable for the Grid environment?• “Cost/profit” analysis / feasibility study

Typical Questions Groups: Current structure of the application Dependencies of the application Available resources (manpower, knowledge, etc.) Requirements for the gridified application Expected impact of gridification Requirements for the grid infrastructure

More info: Application Description Template http://www.lpds.sztaki.hu/gasuc/?m=4


Basic tasks while Porting applications to the Grid (contd.)

3. Grid environment access• Requesting Certificates / VO membership• Accessing Grid environment

• Appropriate VO UI machine account for command line• Portal GUI account;

4. Executing, Testing and Debugging the application;• Testing the non-grid application (debugging in Grid

environment is a hard task), creating use cases for single (non-grid) runs;

5. Constructing the job suite – JDL (Job Description Language) files, executables, auxiliary scripts and input/output data files;

6. Submitting the job to the Grid as small-scale pilot application;


Basic tasks while Porting applications to the Grid (contd.)

7. Executing, Testing and Debugging the pilot application;8. IF something goes wrong

THEN GOTO 4;

9. IF everything seems to workTHEN increase the scale of the application (increase problem size, amount of used resources);

10. Optimizing the grid application;

Grid - korisnicki pristup i razvoj aplikacija

Documents

Transcript of Grid - korisnicki pristup i razvoj aplikacija