JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow
description
Transcript of JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow
![Page 1: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/1.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 1
JIM and JIM and SAMGrid for SAMGrid for
CDFCDF
Mòrag Burgon-LyonMòrag Burgon-Lyon University of GlasgowUniversity of Glasgow
![Page 2: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/2.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 2
ContentsContents
• What is JIM?• What is SAMGrid?• How does JIM relate to SAMGrid?• Components of JIM• Using JIM• Job Types• Station Setup• Deployment Plan
![Page 3: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/3.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 3
What is What is JIMJIM??
Job and Information Management
• Job Management Infrastructure is the framework allowing job submissions for execution on a cluster that has enough resources to complete the job.
• Information Management is knowing what resources are available and the status of the jobs.
![Page 4: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/4.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 4
What is SAMGrid?What is SAMGrid?
• SAMGrid is a grid infrastructure whose goal is to allow globally distributed computing for current experiments, DØ, CDF and Minos at Fermilab.
• Communicating with LHC experiments CMS (Fermilab) and Atlas (Brookhaven)
![Page 5: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/5.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 5
How does JIM relate to SAMGrid?How does JIM relate to SAMGrid?
Complements Sequencial Access via Metadata to provide complete grid services
Job ManagementInformation
and Monitoring
Data Handling
![Page 6: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/6.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 6
How does JIM relate to SAMGrid?How does JIM relate to SAMGrid?
![Page 7: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/7.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 7
How does JIM relate to SAMGrid?How does JIM relate to SAMGrid?
• JIM allows a user to submit jobs to SAMGrid and to access the output files on completion.
• JIM chooses which resources will be used to execute a job. This decision is based on the amount of data required by the job already cached at the sites.
• JIM submits the job to the local BS and SAM provides the I/O data management for the files.
![Page 8: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/8.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 8
How does JIM relate to SAMGrid?How does JIM relate to SAMGrid?
• Condor MMS was expanded for SAMGrid– broker can query SAM station to see how much
data is already present.– dynamic selection of a globus gatekeeper.– the match is determined by calling by calling
external custom code, e.g. in SAMGrid the SAM station is invoked to determine the rank of a match.
• Globus toolkit is used for job transfer and monitoring
![Page 9: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/9.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 9
How does JIM relate to SAMGrid?How does JIM relate to SAMGrid?
• Distinguishes grid level (global) scheduling (selection of a cluster to run) from local scheduling (distribution of the job within the cluster).
• Distinguishes structured jobs (where the details are known to Grid middleware) from unstructured jobs (where the whole job is mapped onto a single cluster).
![Page 10: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/10.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 10
Components of JIMComponents of JIM
• All Sites– sam_gsi_config (includes sam_gridftp and
globus_dh_client and server) for grid security
– xmldb as a SAMGrid interface• Client Site - used to submit jobs to SAMGrid
– typically a remote server or workstation that is used to send jobs to a Submission site
– jim-client
![Page 11: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/11.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 11
Components of JIMComponents of JIM
• Submission Site - maintains a spool of jobs– acts as a client to the broker periodically sending
jobs to available resources at an Execution site– jim_broker_client– www_jim_sandbox (optional)
![Page 12: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/12.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 12
Components of JIMComponents of JIM
• Execution Site - runs the job– sam and sam_station
– sam_batch_adapter
– globus_rm_server – jim_jobmanager– jim_sandbox– jim_config and jim_advertise
![Page 13: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/13.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 13
Components of JIMComponents of JIM
• Monitoring Site - provides information on the state of each submitted job and allows the output of completed jobs to be downloaded– globus_is_server and globus_is_client
– jim_info_providers
![Page 14: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/14.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 14
Using JIM – Submitting a jobUsing JIM – Submitting a job
User creates a jdl file such as the example shown:
sam_dataset = jpmm08-1fileexecutable = retrieve.shinput_sandbox = /home_scotgrid/m/mlyon/test/testjobcpu-per-event = 1sjob_manager = samjob_type = sam_analysissam_universe = prdsam_experiment = cdflog = testjob.logoutput = testjob.outerror = testjob.errarguments = Download Output Yet?group = testinstances = 1
![Page 15: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/15.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 15
Using JIM – Submitting a jobUsing JIM – Submitting a job
• The job is submitted by typing:
samg submit testjobfile.jdf• The progress of the job can be viewed by selecting
the submission site from the list:
http://samgrid.fnal.gov:8080• Select the job from the list. Details of the job state
are displayed• Once the job has been completed the output may
be downloaded
![Page 16: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/16.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 16
Using JIM – Viewing MapUsing JIM – Viewing Map
![Page 17: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/17.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 17
Using JIM – Viewing submission sitesUsing JIM – Viewing submission sites
![Page 18: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/18.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 18
Using JIM – Viewing submitted jobsUsing JIM – Viewing submitted jobs
![Page 19: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/19.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 19
Using JIM – Downloading outputUsing JIM – Downloading output
![Page 20: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/20.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 20
Types of JobsTypes of Jobs
• Monte Carlo– events generated, passed through detector simulation
and reconstructed– e.g. typically no input files, one output file per job– Each job part of a well defined (generator, parameters)
set going to a given dataset
• Reconstruction– real data reconstruction– in general one input file from a dataset going to one
output file in corresponding dataset
• Analysis– an entire dataset is input, many input files
![Page 21: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/21.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 21
Station SetupStation Setup
• Initial installation uses current versions of all products.
• Care must be taken when upgrading – to avoid version incompatibility– to preserve existing configuration
![Page 22: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/22.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 22
What will JIM do once complete?What will JIM do once complete?
• Current functionality allows job submission and output retrieval
• Next steps:– Deployment of secure web-download– User support and defect fixing through testing phase– Roll out of SAMGrid to all CDF sites with available
resources– Add more brokering criteria
• Directors review of Run II computing has recommended the expansion of SAM to be a lab-wide product
![Page 23: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/23.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 23
Deployment PlanDeployment Plan
• Glasgow University has a complete installation of SAMGrid on both ScotGrid and the CDF cluster. This installation is being tested with Monte Carlo simulation.
• Oxford University has SAMGrid installed.
• Installation workshop organised for 20th-22nd Jan 04
![Page 24: JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow](https://reader036.fdocuments.us/reader036/viewer/2022062408/56813ad5550346895da30d6e/html5/thumbnails/24.jpg)
11stst December 2003 December 2003 JIM for CDFJIM for CDF 24
CreditsCredits
• Thanks to the JIM team for providing material for this presentation.