EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows ...

EGEE-II INFSO-RI-031688

Enabling Grids for E-sciencE

www.eu-egee.org

P-GRADE overview and introduction: workflows & parameter sweeps(Advanced features)

Miklos KozlovszkyMTA [email protected]

2



Parameter Sweep applications

• Typical usage of the Grid is that the same application is executed with many different parameters

• P-GRADE portal from release 2.5 supports this functionality

• The next part of the lecture shows how parameter sweep (PS) applications are supported by P-GRADE portal

3



Scaling up a workflow to a parameter study with P-GRADE

Complete workflow

Output file set in an LFC directory (e.g. lfc:/grid/gilda/kozlovszky/myoutputs)

Input file set in an LFC directory (e.g. lfc:/grid/gilda/kozlovszky/myinputs)

4



Turning a WF into a parameter study

By turning at least one of the open input ports into a “PS Input port”

the WF is turned into a Parameter Study WF

6



PS Input Port of Simple PS

Remote file Directory

instead of FILE reference

Do not use the prefix lfn:

if the directory isEGEE Grid file

catalogue

7



Simple PS Activity 2: placement of result

Properties of the VO File Catalog

Menu item PS Properties is available in the Workflow

menu

The Output directory will contain the ZIP-ped

result files of the Workflow executions One SE of the VO

– Choose a reliable one!

8



Executing PS workflows

PS Details for parameter sweep

workflow applications

9



Workflow Manager List PS Details view showing eWF-s

New, middle level list to render the details of a PS Workflow.(Status of complete application)

Statistics shows the progress of the elaboration of the whole PS

The eWorkflow buffer list shows the state of the Workflows being processed.

10



Details view of the eWF Ax_EQU_B_voce_PS.6

Job level details of an eWorkflow

See, that the button Attach is missing as there is not to

much importance to access the WE until the eWorkflow list

is exhausted



www.eu-egee.org

Advanced PS WFs in P-GRADE Portal

12



Advanced parameter studies

Generator component(s)

Initial input data

Generate orcut input into smaller pieces

Collector component(s)

Aggregate result

Files in an LFC directory (e.g. /grid/gilda/kozlovszky/myinputs)

Files in an LFC directory (e.g. /grid/gilda/kozlovszky/myoutputs)

Complete workflow

13



Concept of advanced parameter study workflows

GEN

SEQ

COLL

SEQSEQSEQ

Parameter study part

Collector part evaluates and

integrates the results

Generator part generates the

input parameter space

14



The generic PS workflow application

Third phase:

executing the

collectors

Second phase:

executing the “real”

processor jobs for each

input file combination

First phase: executing

the generators

16



Parameter generator

Generator can be attached to any parameter input port

Generator can be• Auto generator: to generate text files• Custom generator: to generate any content

Generated files are moved into SE by the portal

17



Definition Window of Auto Generator Job

User defines the template of the text file

User puts keys into the template

User defines values for the keys• Integer number• Real number• Custom set• …By clicking on

a key the definition

window for this key is opened.

18



(Auto) Generator Attribute Editor for SE definition

Attribute Editor defines the properties of remote files created by the Generator:

1. Storage Element must be defined if an LCG like (EGEE ) file access has been defined in the PS Output Port belonging to the Generator

19



Detailed view of a PS workflow

Workflow instances

Overall statistics of workflow instances

Collector job(s)

Generator job(s)



www.eu-egee.org

PS practical

21



Matrix multiplication PS10 PS workflows

Multiplication job

1 2 34 5 6

7 8 12

2 1 31 1 13 3 3

Matrix2

1 2 34 5 67 8 9

1 2 34 5 6

7 8 15 . . .

. . .

. . .

22



ExerciseMatrix multiplication PS

Multiplication job

Auto generator

Auto generator

Input files stored on SEs and registered in LFC catalog

1 2 34 5 67 8 9

1 2 34 5 6

7 8 12

1 2 34 5 6

7 8 15

1 2 34 5 67 8 Y

9 <= Y <=36, step 3

2 1 31 1 13 3 3

Matrix2 Output files stored on SEs and registered in LFC catalog

. . .

. . .

. . .

EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows ...

Documents

Transcript of EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE P-GRADE overview and introduction: workflows ...