CCPR WorkshopCCPR WorkshopLexis Cluster IntroductionLexis Cluster Introduction

October 19, 2007October 19, 2007

David Ash David Ash

OutlineOutline What is a cluster? ….two componentsWhat is a cluster? ….two components

Outline Continued…Outline Continued…

How can CCPR’s cluster help you?How can CCPR’s cluster help you? Lexis usageLexis usage

Connecting from WindowsConnecting from Windows Unix commandsUnix commands Submitting jobsSubmitting jobs Managing jobsManaging jobs

Lexis Management SystemLexis Management System Jobs are submitted via cluster Jobs are submitted via cluster

management systemmanagement system An advanced batch system An advanced batch system Efficiently allocates resources to jobsEfficiently allocates resources to jobs Remembers node specs and availabilityRemembers node specs and availability Schedules jobs according to requested Schedules jobs according to requested

resources and priority resources and priority Users can easily view jobsUsers can easily view jobs

How does this help your How does this help your research?research?

Remote access to software and data from Remote access to software and data from anywhere!anywhere!

Incredible machines – faster processors and Incredible machines – faster processors and more memorymore memory

Flexible system to share data and programs Flexible system to share data and programs with colleagueswith colleagues

Opens up your home/office PC for other Opens up your home/office PC for other things, instead of grinding away on a Stata jobthings, instead of grinding away on a Stata job

Advanced users: parallelization, fast compilersAdvanced users: parallelization, fast compilers

Connecting to LexisConnecting to Lexis

Secure Shell SoftwareSecure Shell Software SSH (command line access)SSH (command line access) SFTP (file transferring access)SFTP (file transferring access) Obtain from:Obtain from:http://computing.sscnet.ucla.edu/public/security/http://computing.sscnet.ucla.edu/public/security/

ssh.aspxssh.aspx

Alternative Client SoftwareAlternative Client Software PuTTY (SSH)PuTTY (SSH)

http://the.earth.li/~sgtatham/putty/latest/x86/http://the.earth.li/~sgtatham/putty/latest/x86/putty.exeputty.exe

winSCP (SFTP)winSCP (SFTP)http://winscp.net/eng/download.phphttp://winscp.net/eng/download.php

Connecting to Lexis: Secure Connecting to Lexis: Secure ShellShell

Directions for SSH Secure Shell:Directions for SSH Secure Shell: Click Quick ConnectClick Quick Connect Enter:Enter:

Host Name: lexis.ccpr.ucla.eduHost Name: lexis.ccpr.ucla.edu User Name: your lexis account nameUser Name: your lexis account name Port Number: 22Port Number: 22

Click connect and enter passwordClick connect and enter password Can “Add Profile” for repeat usageCan “Add Profile” for repeat usage Same for SSH Secure File Transfer Same for SSH Secure File Transfer

Mapping Lexis as a Drive using Mapping Lexis as a Drive using SambaSamba

Creating Lexis as a drive in Windows Creating Lexis as a drive in Windows ExplorerExplorer

Must use the VPN to do this action off Must use the VPN to do this action off campus!campus! http://www.bol.ucla.edu/services/vpn/

Instructions I’m going to show are at:Instructions I’m going to show are at: Step-by-step instructions:Step-by-step instructions:

http://lexis.ccpr.ucla.edu/using/samba/http://lexis.ccpr.ucla.edu/using/samba/

Mapping LexisMapping Lexis Open Windows Explorer Open Windows Explorer Select Tools -->Map Network Drive Select Tools -->Map Network Drive At "Folder" text box enter:At "Folder" text box enter:

\\lexis.ccpr.ucla.edu\your-user-name \\lexis.ccpr.ucla.edu\your-user-name

Click "Reconnect at login" if you want the drive Click "Reconnect at login" if you want the drive to be connected each time the computer startsto be connected each time the computer starts

Click the "different user name" link to enter Click the "different user name" link to enter your user name and password for the your user name and password for the connection. connection.

Click "OK" then click the "Finish" button on the Click "OK" then click the "Finish" button on the Map Network Drive window. Map Network Drive window.

Linux CommandsLinux Commands

passwdpasswd Change your Lexis SSH/SFTP passwordChange your Lexis SSH/SFTP password

smbpasswdsmbpasswd Change your Samba passwordChange your Samba password

pwdpwd Print your current directoryPrint your current directory

lsls List files (can take arguments)List files (can take arguments)

ls –lahls –lah L = long, A = all, H = human readableL = long, A = all, H = human readable

cd AND cd ..cd AND cd .. Change directory AND Go up one Change directory AND Go up one directorydirectory

mkdirmkdir foldername foldername Create a directoryCreate a directory

cp cp file1 file2file1 file2 Copy Copy file1file1 to to file2file2

mv mv file1 file2file1 file2 Rename/Move Rename/Move file1file1 to to file2file2

more more file file OROR catcat filefile

Display Display filefile to screen to screen

du –h du –h Disk Usage (-h = human readable)Disk Usage (-h = human readable)

tar –cvf AND tar -tar –cvf AND tar -xvfxvf

Compress a file/directory / Expand .tarCompress a file/directory / Expand .tar

gzip AND gunzipgzip AND gunzip Compress a file (.gz extension) / Expand .gzCompress a file (.gz extension) / Expand .gz

ln –s ln –s actual linkactual link Creates a symbolic link/shortcutCreates a symbolic link/shortcut

Getting Help in LinuxGetting Help in Linux

man man commandcommand list help for list help for commandcommand

man –k man –k commandcommand keyword search for keyword search for commandcommand

whatis whatis commandcommand brief description of brief description of commandcommand

which which namename search for search for namename

aproposapropos keyword keyword list commands with list commands with keywordkeyword in their in their descriptiondescription

helpdesk@ccpr.ucla.edhelpdesk@ccpr.ucla.eduu

Bryan and myself. Bryan and myself.

Linux File EditorsLinux File Editors

PicoPico – easiest editor. Open and type. – easiest editor. Open and type. Commands listed at bottom of editor.Commands listed at bottom of editor.

ViVi – more advanced. You can move – more advanced. You can move around quickly, do copy/paste, around quickly, do copy/paste, search, etc, but keystrokes required search, etc, but keystrokes required for everything. for everything.

EmacsEmacs – most advanced – most advanced OR edit files over Samba from OR edit files over Samba from

Windows.Windows.

Submitting Stata and SAS Submitting Stata and SAS jobsjobs

Submit jobs to the Cluster using ONE of Submit jobs to the Cluster using ONE of the following commands: qjob, qstata, the following commands: qjob, qstata, qsas, qmatlabqsas, qmatlab

Will be prompted for: Will be prompted for: Job type (for qjob only!)Job type (for qjob only!) FilenameFilename Amount of Memory (number in megs, ex. 1050)Amount of Memory (number in megs, ex. 1050) If memory > 1 gig --> Over 2 hours (y/n)If memory > 1 gig --> Over 2 hours (y/n) Confirmation to submit (y/n)Confirmation to submit (y/n)

Quick Method:Quick Method: qstata sample.do 500 y qstata sample.do 500 y

Qstata example outputQstata example output

Two Logs are createdTwo Logs are created filename.logfilename.log - This is the log file from Stata. The - This is the log file from Stata. The

output of your entire Stata job is captured in this output of your entire Stata job is captured in this file.file.

filename.do.joblogfilename.do.joblog - This file will capture the - This file will capture the output of your cluster command file and any output of your cluster command file and any errors that may occur (Stata output is NOT errors that may occur (Stata output is NOT captured by this log.)captured by this log.)

filename.do.cmdfilename.do.cmd - This is the command file that is - This is the command file that is submitted to the cluster.submitted to the cluster. Quick TIP: qsub filename.do.cmdQuick TIP: qsub filename.do.cmd

Cluster job monitoringCluster job monitoring

qstatqstat or or qstatusqstatus to view overall to view overall cluster usagecluster usage

qstat –jqstat –j jobnumber jobnumber for information for information particular to your jobparticular to your job

qdel qdel jobnumberjobnumber to kill your job to kill your job

Interactive Sessions Interactive Sessions

Interactive UsageInteractive Usage Meant for testing before submitting to Meant for testing before submitting to

clustercluster Limited spotsLimited spots Logoff promptly when finishedLogoff promptly when finished

Non-GUI Interactive SessionNon-GUI Interactive Session Type “qinteractive”, can also specify Type “qinteractive”, can also specify

memory after command. Default 256mb.memory after command. Default 256mb. GUI Interactive SessionGUI Interactive Session

Login with XliveCDLogin with XliveCD Type “qlive” – Run sas/matlab/xstataType “qlive” – Run sas/matlab/xstata

Questions / Questions / Comments?Comments?