Introduction to the Unity Cluster...1. pwd –printcurrentlocation 2. ls...

Introduction Connecting to the Unity Cluster Transferring Files Commands for Navigating the Cluster Programs for Editing and Running Code

Introduction to the Unity Cluster

Jason Thomas

June 26, 2018

Jason ThomasIntroduction to the Unity Cluster


Introduction



Motivation: Why use the Unity Cluster

I SpeedI processing speed(?); CPU cache; free up resources on local

machine for long running jobsI Large data setsI Remote accessI Stepping stone for Ohio Super Computer (spark cluster

computing system)



Tools Needed for Getting up and Running

1. Connect to the Unity Cluster2. Transfer Files3. Commands for navigating the cluster4. Programs for editing and running (Stata/R) code



Background

I Unity cluster runs linuxI Clusters & NodesI Interactive Sessions & adding jobs to the queue



Connecting to the Unity Cluster



VPN: Virtual Private Network

How do we connect: Step 1

I If connecting from an off-campus location, we need to establisha connection via a VPN

I Software recommended by OSU:I Cisco AnyConnect Secure Mobile Client:I anyconnect-windows-4.5 (.exe for Windows; .dmg for Mac)I https://osuasc.teamdynamix.com/TDClient/KB/ArticleDet?

ID=14542I Ready to connect. vpn.asc.ohio-state.edu -> ConnectI Username & Password: same as emailI Second Password: Duo Mobile


https://osuasc.teamdynamix.com/TDClient/KB/ArticleDet?ID=14542

https://osuasc.teamdynamix.com/TDClient/KB/ArticleDet?ID=14542


X Windows System


Enable the Unity Cluster to open new windows on your computer

I X Windows SystemI XWin32 & FastX are alaible for OSU faculty and students

I Xming what we will use today (available from Sourceforge)I popular X Win server for WindowsI free and in the public domain (under MIT, BSD, and other

licenses)


https://ocio.osu.edu/software/directory/slwin#xwin32

https://sourceforge.net/projects/xming/?source=typ_redirect


SSH: Secure Socket Shell


I The Unity cluster uses the linux operating systemI interaction through a terminal (typing commands at a prompt)

I Software for connecting to linux servers: PuTTYwww.chiark.greenend.org.uk/~sgtatham/putty/latest.htmlI (download, install, and run from the start menu: PuTTY

Desktop App)I Configuration:

I Host Name: unity.asc.ohio-state.eduI Saved Sessions: Unity (or whatever you want to name it) ->

SAVEI Connection -> SSH -> X11 -> Enable X11 forwarding (Session

-> SAVE!)


https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html


Transferring Files



SFTP: Secure File Transfer Protocol

How do we upload (download) files to (from) the UnityCluster?

I PuTTY also has SFTP Client (secure file transfer protocol)I not as friendly, so we’ll use a different tool. . .

I FileZilla has a graphical user interface (GUI)I (Remember that we need to connect through a VPN)I Configuration:

I Host: : unity.asc.ohio-state.eduI Username & Password: same as OSU emailI Port: 22


https://filezilla-project.org/download.php?show_all=1


Commands for Navigating the Cluster



7(ish) Commands for the linux terminal

Inside the PuTTY window. . .

1. pwd – print current location2. ls – print contents of current folder3. mkdir NAME – make a new folder called NAME in the current

location4. cd PATH – change to new location PATH5. rm Filename – remove the file Filename

I rmdir – remove empty folder6. mv Filename newPATH – move the file Filename to the

location newPATH7. exit – does just that



Tips for the linux terminal


I cd – without a path will take you to your home directoryI TAB – complete file/folder namesI TAB TAB – offer suggestions for possible file/folder namesI cp -r FOLDER newPATH – copy a folder (and its contents)

to newPATHI ls -la – prints more information about folder contents (and

lists files with names starting with a period, e.g., .bashrc)I You can set up aliases in the file .bashrc

I alias data = ’cd /Research/Data/; ls -l’I Up Arrow – cycles through previous commands



Programs for Editing and Running Code



Editing Files


I gedit – basic text editorI emacs – powerful text editor with a steep learning curve

I great for writing code (ESS – Emacs Speaks Statistics)I other options: vi and nano



Running Stata


List available programs> module avail

I spacebar – page down; q – quit

Start interactive session for 2 hours> qsub -I -l walltime=2:00:00



Running Stata (cont.)

starts an interactive session (for default 1 hour) using 1 node, 4processors, and 16Gigs of RAM> qsub -I -l nodes=1:ppn=4,mem=16GB}

Once the session starts (it may take a while depending on traffic)> module load stata

> stata



Running R

First we need to load the intel module> module load intel

List available versions of R then load and start R> module avail R/

> module load R/3.5.0

> R

> 3 + 5> q('no')



Running R BATCH Job

I Batch files can be submitted using the Portable Batch System(PBS)

I Be sure to avoid DOS/Windows line endingsI lines must end with a linefeed (LF)I tabs must be 4 spaces

I See example PBS script: run_R_job.pbsI Submit the R BATCH job with the following command

> qsub -d "$PWD" run_R_jobs.pbs

the -d "PWD" option starts R in the current directory as opposed tothe home directory (the default)



Useful Commands for the qsub system

I qstat -u username.number – querry your current jobsI showq – print the current queueI qdel jobID – delete job with ID jobID

I qstat will list the job IDI for more see OSC’s page


https://www.osc.edu/supercomputing/batch-processing-at-osc/monitoring-and-managing-your-job

Introduction to the Unity Cluster...1. pwd –printcurrentlocation 2. ls...

Documents

Transcript of Introduction to the Unity Cluster...1. pwd –printcurrentlocation 2. ls...