Post on 30-Jun-2019
3/3/17
1
Parallel MATLAB on HPC
AvalonJohnson,JimiChui,CesarSulandErinShaw
USCCenterforHigh-PerformanceCompu6ng(HPC)HPCResearchandEduca6onFacilita6on
USC HPC§ Founda;onforcomputa;onalresearchatUSC
• Sharedresourcethatisfreelyavailabletofaculty,researchersandgraduatestudents
• Worldclasssupercompu;ngcenter- Currentlyranked12thfastestacademiccluster
§ Research&Educa;onFacilita;on• Drop-intoofficehoursorrequestaconsulta;on• AKendanewusermee;ngorpar;cipateinaworkshop
- Seeh6ps://hpc.usc.edu/educa;on- Seeh6ps://hpc.usc.edu/officehours- Emailhpc@usc.edu
3/3/17
2
Outline§ Introduc;on
• Prerequisites,setupworkspaceandlaunchMATLAB• Introduc;ontoparallelMATLAB
§ RunMATLABonsinglenodew/ParallelCompu;ngToolbox(PCT)• Usinggraphicaluserinterface(GUI)• Usingcommandlineinterface(CLI)• UsingCLIwithbatchparalleljobexecu;on
§ RunMATLABonmul;plenodesw/DistributedCompu;ngServer(DCS)• Usingcommandlineinterface(CLI)
§ Beyondparfor• labIndex,Composite,andBroadcast&Receive• Prac;ceusingPBS
Outline§ Introduc6on
• Prerequisites,setupworkspaceandlaunchMATLAB• Introduc;ontoparallelMATLAB
§ RunMATLABonsinglenodew/ParallelCompu;ngToolbox(PCT)• Usinggraphicaluserinterface(GUI)• Usingcommandlineinterface(CLI)• UsingCLIwithbatchparalleljobexecu;on
§ RunMATLABonmul;plenodesw/DistributedCompu;ngServer(DCS)• Usingcommandlineinterface(CLI)
§ Beyondparfor• labIndex,Composite,andBroadcast&Receive• Prac;ceusingPBS
3/3/17
3
Prerequisites§ YouwillneedX11DisplayForwardingtousetheGUI
• OnWindows:Set‘EnableX11forwarding’onyourclient(thisisusuallythedefault)
• OnMacs:DownloadandinstallXquartz(www.xquartz.org)
§ Login
§ Run‘xeyes’programtoverifythatdisplayisworking
$ ssh -X {USCNetId}@hpc-login3.usc.edu
$ xeyes #now move cursor$ ^C #control-c to quit
Workspace setup§ Alwaysworkinyourprojectdirectory!
§ Createamatlabworkshopdirectory©workshopfilestoit
§ Youshouldnowhavethefollowingfiles
$ mkdir matlab$ cd matlab$ cp /home/rcf-proj/workshop/matlab/* .
$ cd /home/rcf-proj/{project}/{user}
$ lsbroadcastReceive.mEs;matePi.mjobScript.mlabIndex.msubmit.pbssubmitTasks.m
3/3/17
4
Request an interactive compute node
-l procs=8Use“procs”insteadoftypical“nodes=x:ppn=x”DistributesMATLABprocessesacrossnodesmostefficiently
-l software=MPCT+1 Checkout1MATLABPCTlicenseforinterac;veapplica;on(client)Eachcoreneedsalicense–MATLABwillrequestmorelicensesperourconfigura;on
-l advres=HPCCWorkshop.xx –A workshopUsenodesreservedinadvanceforthisworkshopOmittheselinesaXerworkshop
NOTE:Whenyourunrealjobs,ifyouhaveaccesstoacondoqueue,besuretoincludethe–q<queue_name>op;on
$ qsub -I l walltime=02:00:00 –l procs=8 –l software=MPCT+1 -A workshop -l ‘advres=HPCWorkshop.xx’
[csul@hpc-login3 ~]$ qsub I -l 'walltime=1:00:00’-l 'procs=8’ –l 'software=MPCT+1’ qsub: waiting for job 14526397.hpc-pbs.hpcc.usc.edu to start qsub: job 14526397.hpc-pbs.hpcc.usc.edu ready ---------------------------------------- Begin PBS Prologue Tue Apr 5 13:33:13 PDT 2016 Job ID: 14526397.hpc-pbs.hpcc.usc.edu Username: csul Group: ucsadmin Project: default Name: STDIN Queue: quick Shared Access: no All Cores: no Has MIC: no Nodes: hpc2289 TMPDIR: /tmp/14526397.hpc-pbs.hpcc.usc.edu End PBS Prologue Tue Apr 5 13:33:13 PDT 2016 ----------------------------------------
Receive an interactive compute node
3/3/17
5
Launch MATLAB in second window
Thisisyouroriginalhpc-loginshell
When‘qsub’succeeds,acomputenodeshellwillopeninthiswindow
DonotrunMATLABwithGUIinthisshell,itwillbeveryslow;runinnewterminalwindowinstead
Usethiswindowtoedityourprogramandmonitoryourjob
Openasecondterminalwindowandlogintoheadnode(with–X) $ ssh –X hpc-login3.usc.edu
ThenlogintocomputenodeandrunMATLABfromhere $ ssh –X hpcxxxx $ source /usr/usc/matlab/
R2016a/setup.sh $ matlab&
Outline§ Introduc6on
• Prerequisites,setupworkspaceandlaunchMATLAB• Introduc6ontoparallelMATLAB
§ RunMATLABonsinglenodew/ParallelCompu;ngToolbox(PCT)• Usinggraphicaluserinterface(GUI)• Usingcommandlineinterface(CLI)• UsingCLIwithbatchparalleljobexecu;on
§ RunMATLABonmul;plenodesw/DistributedCompu;ngServer(DCS)• Usingcommandlineinterface(CLI)
§ Beyondparfor• labIndex,Composite,andBroadcast&Receive• Prac;ceusingPBS
3/3/17
6
How MATLAB supports parallelization§ Clusterobjects
parallel.cluster.Local( interactwithclusteronlocalmachine)parallel.cluster.Torque(interactwithclusterrunningTorque)parallel.cluster.Mpiexec(interactwithclusterusingmpiexec)
§ ParallelCompu;ngToolbox(USClicensed)parpool+parfor
§ DistributedCompu;ngServer(USClicensed)batchparfor,spmd(singleprogram,mul;pledata)jobs,spmdMPIcommunica;on
MATLAB’s parallel machinery Loginnode
3/3/17
7
MATLAB program: EstimatePi.m% Calculate the value of Pi using a Monte Carlo simulation max=1e9; tic; n=0; parfor i = 1:max
x=rand; y=rand; if (x^2 + y^2 < 1.0)
n=n+1; end
end elapsedTime = toc; pi = (4.0 * n / max);
MATLAB’s parfor loop
3/3/17
8
Outline§ Introduc;on
• Prerequisites,setupworkspaceandlaunchMATLAB• Introduc;ontoparallelMATLAB
§ RunMATLABonsinglenodew/ParallelCompu6ngToolbox(PCT)• Usinggraphicaluserinterface(GUI)• Usingcommandlineinterface(CLI)• UsingCLIwithbatchparalleljobexecu;on
§ RunMATLABonmul;plenodesw/DistributedCompu;ngServer(DCS)• Usingcommandlineinterface(CLI)
§ Beyondparfor• labIndex,Composite,andBroadcast&Receive• Prac;ceusingPBS
Run on single node with PCT (GUI)MATLAB’s Parallel Computing Toolbox (MPCT)
§ Insecondterminalwindow
x
§ TherearethreestepsforconfiguringMATLAB’sPCT1.Setclusterprofile(SingleNodeProfile_GUI)
TellMATLABwhichPBSresourcesareavailable
2.SetparallelpoolpreferencesTellMATLABwhichclusterprofileandhowmanyworkerstouseforthisrun
3.Startparallelpoolandexecuteprogram
§ WhenMATLABready,setpathtoyourmatlabdirectory
$ ssh -X hpcxxxx$ source /usr/usc/matlab/R2016a/setup.sh #add matlab to path$ matlab
3/3/17
9
1. Set cluster profile (GUI)
3/3/17
10
3/3/17
11
§ Set“Descrip;on”(op;onal)§ Set“NumWorkers”
#workers=#cores(procs)§ Set“JobStorageLoca;on”
whereresultswillgo§ Closewindow
2. Set parallel pool preferences (GUI)
§ AccessPreferencespanelfromtoptaborlower-lepbuKon§ Set“DefaultCluster”
selectaprofiletouse§ Set“Preferrednumberofworkersinaparallelpool”
#workers=#cores–1(i.e.,leaveonecoreforMATLAB)§ Closewindow
3/3/17
12
3. Start parallel pool (GUI)§ Select“Startparallelpool”
• Parallelbarswillblinkun;lready• Hoveroverforrunstatus
§ In“CurrentFolder”window• Selectscript(es;matePI.m)• OpensinEditorwindow
§ Runprogram(greentriangle)• Type$toptoseerunningprocesses
§ ExitMATLAB
Outline§ Introduc;on
• Prerequisites,setupworkspaceandlaunchMATLAB• Introduc;ontoparallelMATLAB
§ RunMATLABonsinglenodew/ParallelCompu6ngToolbox(PCT)• Usinggraphicaluserinterface(GUI)• Usingcommandlineinterface(CLI)• UsingCLIwithbatchparalleljobexecu;on
§ RunMATLABonmul;plenodesw/DistributedCompu;ngServer(DCS)• Usingcommandlineinterface(CLI)
§ Beyondparfor• labIndex,Composite,andBroadcast&Receive• Prac;ceusingPBS
3/3/17
13
CLI: Run on single node with PCT§ Next,we’lltakethesamethreestepsusingcommandline
• Remember,weares;llrunninginterac;velyonacomputenode
§ StartMATLABwithoutGUI
$ source /usr/usc/matlab/R2016a/setup.sh$ matlab –nodesktop -nodisplay
MATLAB is selecting SOFTWARE OPENGL rendering. < M A T L A B (R) > Copyright 1984-2015 The MathWorks, Inc. R2015a (8.5.0.197613) 64-bit (glnxa64) February 12, 2015 To get started, type one of these: helpwin, helpdesk, or demo. For product information, visit www.mathworks.com.
Academic License >>
1. Set cluster profile (CLI)§ Createclusterprofileforsingle-nodeparalleljob
• TellsPCTwhichPBSresourcesareavailable
• Setjobworkdirectory(singlequotesrequired)
• Setnumberofworkerstoqsubprocsvalue(PCTmax=12)
>> MPCTprofile = parallel.cluster.Local
>> MPCTprofile.JobStorageLocation = '/home/rcfproj/{project}/{user}/{matlab_dir}'
>> MPCTprofile.NumWorkers = 8
3/3/17
14
Set cluster profile§ Youcansaveyourprofileandreuseit
• Thisprofileissavedin.matlab,andisre-usable• Ifyouhavealreadycreatedthis,next;meyourun,insteadofalltheformercommands,justsay:
§ Atthispointwecaneither• RunEs;matePi.mscriptinterac;velyintheforegroundorrunitinthebackgroundasabatchjob
>> MPCTprofile.saveAsProfile('SingleNodeProfile')
>> MPCTprofile = parcluster('SingleNodeProfile')
2. Set parallel pool and execute program%Start parpool (specify cluster profile and workers)>> parpool(MPCTprofile,7)
Starting parpool ….connected to 7 workers. %Run program >> estimatePi
ans = Elapsed time = …...
%Check values >> pi
pi = 3.1416
%Always clean up >> delete(gcp(‘nocreate’))
3/3/17
15
Outline§ Introduc;on
• Prerequisites,setupworkspaceandlaunchMATLAB• Introduc;ontoparallelMATLAB
§ RunMATLABonsinglenodew/ParallelCompu6ngToolbox(PCT)• Usinggraphicaluserinterface(GUI)• Usingcommandlineinterface(CLI)• UsingCLIwithbatchparalleljobexecu6on
§ RunMATLABonmul;plenodesw/DistributedCompu;ngServer(DCS)• Usingcommandlineinterface(CLI)
§ Beyondparfor• labIndex,Composite,andBroadcast&Receive• Prac;ceusingPBS
MATLAB batch parallel job execution§ YoucanspawnjobsfromwithinMATLAB
• Abatchjobwillruninthebackground§ MATLABhasitsownschedulerforbatchjobs
• Forexample,ifyouhadtwoindependenttasks
§ RunabatchjobusingPCT>> job = batch(MPCTprofile, 'EstimatePi', ’pool', 7)
3/3/17
16
MATLAB batch parallel job execution§ Toviewjobstate
§ Toviewtheresults
§ Typevariablenamestoseeresults
>> job %view state of job
>> pi pi = 3.1416
>> elapsedTime elapsedTime = 1.9440
>> load(job)
Outline§ Introduc;on
• Prerequisites,setupworkspaceandlaunchMATLAB• Introduc;ontoparallelMATLAB
§ RunMATLABonsinglenodew/ParallelCompu;ngToolbox(PCT)• Usinggraphicaluserinterface(GUI)• Usingcommandlineinterface(CLI)• UsingCLIwithbatchparalleljobexecu;on
§ RunMATLABonmul6plenodesw/DistributedCompu6ngServer(DCS)• Usingcommandlineinterface(CLI)
§ Beyondparfor• labIndex,Composite,andBroadcast&Receive• Prac;ceusingPBS
3/3/17
17
Run on multiple nodes (GUI)MATLAB’s Distributed Computing Server (MDCS)
§ Requesttwonodes
§ Opennewwindow
f
§ TherearethreestepsforMATLABMDCSconfigura;on1.Setclusterprofile(Mul;NodeProfile_GUI)
Setupresourcetemplate:TellMATLABhowtopassresourcestoPBS
2.SetparallelpoolpreferencesTellMATLABwhichclusterprofileandhowmanyworkerstouseforthisrun
3.Startparallelpoolandexecuteprogram
$ ssh –X {user_id}:hpc-login3.usc.edu$ ssh -X hpcxxxx$ matlab
qsub -I l walltime=01:00:00 –l nodes=2:ppn=8 –l software=MPCT+1 -A workshop -l ‘advres=HPCWorkshop.xx’
§ OpenMainMenu=>Environment=>Parallel=>ManageClusterProfiles§ ChooseTorque§ Renameto“Mul;NodeProfile_GUI”§ Edit
1. Set cluster profile
3/3/17
18
Set resource template
§ SetJobStorageLoca;on§ SetNumWorkers§ SetResourceTemplate
-lprocs=^N^-sopware=MDCS+^N^-mabe–M{user_id}@usc.edu
2. Set parallel pool preferences
§ AccessPreferencespanelfromtoptaborlower-lepbuKon§ Set“DefaultCluster”§ Set“Preferrednumberofworkersinpool”
#workers=#cores–1§ Closewindow
3/3/17
19
3. Start parallel pool§ Select“Startparallelpool”
• Parallelbarswillblinkun;lready• Selectscript(Es;matePI.m)andrun
§ Howtocheckyourmul;nodeprocess?• $myqueue
- Onejobisourstdinjob(interac;veshell)- Onejobismatlab’swhichhasrequested15cores(andreceivedtwonodes)andiswai;ng
- Youcansshtoanyoneofthesenodesandrun$top
§ ExitMATLAB
Outline§ Introduc;on
• Prerequisites,setupworkspaceandlaunchMATLAB• Introduc;ontoparallelMATLAB
§ RunMATLABonsinglenodew/ParallelCompu;ngToolbox(PCT)• Usinggraphicaluserinterface(GUI)• Usingcommandlineinterface(CLI)• UsingCLIwithbatchparalleljobexecu;on
§ RunMATLABonmul6plenodesw/DistributedCompu6ngServer(DCS)• Usingcommandlineinterface(CLI)
§ Beyondparfor• labIndex,Composite,andBroadcast&Receive• Prac;ceusingPBS
3/3/17
20
Run on multiple nodes (CLI)§ Next,we’lltakethesestepsusingthecommandline§ StartMATLABwithoutGUI
$ source /usr/usc/matlab/R2016a/setup.sh$ matlab –nodesktop -nodisplay
MATLAB is selecting SOFTWARE OPENGL rendering. < M A T L A B (R) > Copyright 1984-2015 The MathWorks, Inc. R2015a (8.5.0.197613) 64-bit (glnxa64) February 12, 2015 To get started, type one of these: helpwin, helpdesk, or demo. For product information, visit www.mathworks.com.
Academic License >>
Setting MDCS profiles
§ CreateclusterprofileforMATLAB’sDCS• TellsMATLABhowtotalktomul;plenodes
§ Saveprofile,andreuselater
>> MDCSprofile = parallel.cluster.Torque >> MDCSprofile.JobStorageLocation =
'/home/rcfproj/{project}/{user}/{matlab_dir}'
>> MDCSprofile.saveAsProfile(’MultiNodeProfile')
>> MPCTprofile = parcluster(’MultiNodeProfile')
3/3/17
21
Setting MDCS profiles
§ SetResourceTemplate
§ Note:Thereisnowall;mespecified§ Nwillbedeterminedbyuslater
>> MDCSprofile.ResourceTemplate=' -l procs=^N^ -l software=MDCS+^N^ -m abe -M {user_id}@usc.edu'
Setting MDCS profiles
§ MATLABalsoneedstoknow• Howtologintocomputenodes(ssh)• Howtotransferfiles(scp)
§ Saveyourprofile
>> MDCSprofile.RshCommand='ssh' >> MDCSprofile.RcpCommand='scp'
>> MDCSprofile.saveAsProfile('MultiNodeProfile')
3/3/17
22
Using MDCS profiles§ Usingyourpreviouslysavedprofile
§ Setmaximumwall;me
§ Ifyouhaveawall;meyouuseopenyoucansaveandrestoretheprofile
>> MPCTprofile = parcluster('OneHr_MultiNodeProfile ')
>> MDCSprofile.SubmitArguments=’ l walltime=1:00:00'
>> MDCSprofile.saveAsProfile('OneHr_MultiNodeProfile')
>> MDCSprofile = parcluster('MultiNodeProfile')
Launching an MDCS job§ Howmanycorestorequest?
• Mostnodesinmainqueuehaveeither8or16corespernodesomul;plesof8(8,16,24,32…)or16(16,32,48…)arebest
• Makesiteasierforjobschedulertofindyourresources,jobstartssooner
§ Launchabatchjob
• 16cores,1forMATLABplus15workers• MATLABhandlesqsubdetails• NOTE:YoucancloseyourMATLABsessionandinterac;vesessionatthispointifyouaresavingresultstofile(butdon’tdofortutorial)
• Check$myqueue
>> job = batch(MDCSprofile,'EstimatePi', 'parpool', 15)
3/3/17
23
Multinode job: Checking status>> jobs=findJob(MDCSprofile) jobs= 8x1 Job array: ID Type State FinishTime Username Tasks -------------------------------------------------------------- 1 5 independent running csul 1 2 6 pool finished Apr 18 11:01:49 csul 7 3 7 pool finished Apr 22 15:05:07 csul 3 4 8 pool finished May 27 10:04:36 csul 32 5 9 pool finished May 27 10:04:27 csul 32 6 10 pool finished May 27 10:35:18 csul 32 7 11 pool finished May 27 10:36:23 csul 32 8 12 pool finished May 27 11:39:00 csul 32
Multinode job: Retrieving results
>> load(jobs(12)) >> pi pi =
3.1416 >> elapsedTime elapsedTime =
42.3374
3/3/17
24
Outline§ Introduc;on
• Prerequisites,setupworkspaceandlaunchMATLAB• Introduc;ontoparallelMATLAB
§ RunMATLABonsinglenodew/ParallelCompu;ngToolbox(PCT)• Usinggraphicaluserinterface(GUI)• Usingcommandlineinterface(CLI)• UsingCLIwithbatchparalleljobexecu;on
§ RunMATLABonmul;plenodesw/DistributedCompu;ngServer(DCS)• Usingcommandlineinterface(CLI)
§ Beyondparfor• labIndex,Composite,andBroadcast&Receive• Prac6ceusingPBS
Practice using PBS§ Thereareprac;cescriptsin/home/rcf-proj/workshop/matlab
• Copythemtoyourmatlabworkshopdirectory
§ Normallyyouwoulddothefollowing1. CreateaPBSscript2. Submitabatchjobfromheadnode(e.g.,qsubsubmit.pbs)
§ Youcantrythisnow- Exitcomputenode(youshouldbeonheadnode)- Thescriptssubmit.pbsandjobScript.mshowhowtosubmitEs;matePito
thequeue
3/3/17
25
Submit EstimatePi.m to queue
#!/bin/bash#PBS-lnodes=1:ppn=1#Launchmatlab,workerswillruninanotherjob#PBS-Nmatlab_launchersource/usr/usc/matlab/R2016a/setup.shcd$PBS_O_WORKDIRmatlab-nodisplay-nosplash-nodesktop-r"jobScript"
parpool('Mul;NodeProfile_GUI',7)%Increaseworkersasdesired%ReplacewithotherexamplesEs;matePi%IncreasesizeofproblemasdesiredpielapsedTimedelete(gcp('nocreate'))
submit.pbs jobScript.m
Beyond parfor§ parforisthemostbasicwaythatMATLABcan
parallelizeyourcode§ Some;mesyou'llneedtohavemorecontrolover
whatgetsrunoneachworker§ Herearesomeexamplestogiveyouanideaofthe
capabili;esofMATLAB’sParallelCompu;ngToolboxandDistributedCompu;ngServer
3/3/17
26
Beyond parfor§ labIndex§ Composite§ Broadcast&receive§ Lookatthescriptsyoucopiedfrom/home/rcf-proj/
workshop/matlab§ Seeh6p://www.mathworks.com/help/distcomp/execute-
simultaneously-on-mul;ple-data-sets.html?requestedDomain=www.mathworks.com
submitTasks.m %Demotasksubmission%Thisscriptshowyouhowtocreatejobandtask%andsubmitthem%loadthedefaultclusterprofilec=parcluster;%Youcanspecifyadifferentprofileifyoulike%c=parcluster('Mul;NodeProfile_CLI')%createsubmitargumentsifyouwanttorunthisasamul;nodejob%c.SubmitArguments='-lwall;me=00:30:00-E';%createjobj=c.createJob;a=rand(4);
%createtwotasksj.createTask(@sin,1,{a});j.createTask(@cos,1,{a});%runbothtasksatthesame;mej.submit;%wait;lljobfinishbeforefetchingresultsj.wait('finished');pause(20);%fetchoutputresult=j.fetchOutputs;f=[result{:}];%savedatasave('submitTasks.out','f','-ascii');
3/3/17
27
labIndex.m %Demoofparpool,Compositeandlabindex
%ThisscriptdemosparpoolandComposit,labindex%itmustberunasamul;-nodejob%loadthedefaultclusterprofile%c=parcluster;%Youcanspecifyadifferentprofileifyoulikec=parcluster('Mul;NodeProfile_CLI')%createsubmitargumentsc.SubmitArguments='-lwall;me=00:30:00-E';%request3workersc.parpool(3)
%computeprocessa=Composite();spmdtemp=labindex*ones(10);a=temp*temp;end
%saveworkerresultsd=[a{:}];Save(’labIndex.out','d','-ascii');%cleanupdelete(gcp('nocreate'))
broadcastReceive.m %Demolabindexandmpicommunica6on
%demolabindexandmpicommunica;on,send,receive,broadcast%Thisscriptmustberunasamul;-nodejob%loadthedefaultclusterprofile%c=parcluster;%Youcanspecifyadifferentprofileifyoulikec=parcluster('DefaultMul;NodeProfile')%createsubmitargumentsc.SubmitArguments='-lwall;me=00:30:00-E';c.parpool(3)
%Setupa'singleprogram,mul;pledata'environmentspmdswitchlabindex%Iflabindexis1,1:5toscreen%iflabindexis2,receivedatafrom1onwhattoprinttoscreen%otherwiseprint11:15toscreencase1A=1:5;labSend(A,2,199);%sendtoworker2case2A=labReceive(1,199);%receivefromworker1otherwiseA=11:15;%createhisownendendA{:}1/3 2/3
3/3/17
28
broadcastReceive.m spmd%iflabindexis1,print6:10toscreen%otherwisereceivedatafrom1onwhattoprinttoscreeniflabindex==1A=labBroadcast(1,6:10);%broadcasttoallelseA=labBroadcast(1);%receivebroadcastendendA{:}delete(gcp('nocreate')) 3/3
End Notes§ MATLABdocumenta;on
h6p://www.mathworks.com/help/
§ MATLABlicenses• Toviewavailability,typeoncommandline/auto/usc/matlab/R2016a/etc/glnxa64/lmu;llmstat-a-c27000@hpc-licenses.usc.edu|grepMATLAB
• ToviewgraphicallyhKps://hpc-monitoring.usc.edu/dashboards/queues.html