Introduction to Linux and the Math Department...

Introduction to Linux and the Math DepartmentComputers

David Love

Software Interest GroupUniversity of ArizonaSeptember 24, 2012

1 Introduction

2 BasicsConnecting to UNIX/LinuxBasic CommandsLinux Filesystem

3 More Advanced CommandsA List of Useful CommandsFile Transfer with LinuxPermissions and Executables

4 Your Environment, Shell Scripting & Variables

5 Making Things Better & Easier

Intro

What Are UNIX and Linux?

UNIX is an operating system originally developed in 1969 at BellLabs.

Has been continually modified since then.

Currently, UNIX is a set of standards that an OS might abide by

Apple OSX is UNIX.Technically, Linux is not UNIX.Sometimes see “*nix” to denote both UNIX and “Unix-like”systems.

Linux is an open source operating system, originally developed in1992.

Has been under continual development, is still free and opensource.

David Love Linux and Math September 24, 2012 1 / 54

Basics

Why use UNIX/Linux?

It is the OS of high-performance computing.

As of June 2012, 462 of the 500 fastest supercomputers ran Linux.1

25 run some other UNIX system.11 are listed as “mixed”

Which leaves 2 that run Windows.

Lots of academic software is written for Linux.

Run code without using up your computer.

Easy file transfer

Backups!Always have homework available.

1http://top500.org/stats/list/37/osDavid Love Linux and Math September 24, 2012 2 / 54

http://top500.org/stats/list/37/os

Basics

Why use UNIX/Linux?

It is the OS of high-performance computing.

As of June 2012, 462 of the 500 fastest supercomputers ran Linux.1

25 run some other UNIX system.11 are listed as “mixed”Which leaves 2 that run Windows.

Lots of academic software is written for Linux.

Run code without using up your computer.

Easy file transfer

Backups!Always have homework available.

1http://top500.org/stats/list/37/osDavid Love Linux and Math September 24, 2012 2 / 54

http://top500.org/stats/list/37/os

Basics Connection

Connecting to Math Linux

Windows Three options:

UA Software https://sitelicense.arizona.edu/ssh/

PuTTY http://www.chiark.greenend.org.uk/

~sgtatham/putty/

Cygwin http://www.cygwin.com/

Apple Comes with a terminal.

Linux Comes with a terminal.


https://sitelicense.arizona.edu/ssh/

http://www.chiark.greenend.org.uk/~sgtatham/putty/

http://www.chiark.greenend.org.uk/~sgtatham/putty/

http://www.cygwin.com/

Basics Connection

Where do we Connect?

Math & Applied Math (& Statistics?) grad students will connect to:

gila.math.arizona.edu Main server for doing most everything.

iguana.math.arizona.edu Server for working with remote desktop.

chivo1,chivo2,chivo3,chivo4 For high performance computing.MUST first connect through another server (gila).

password For changing your department password. Use commandyppasswd. Seemath.arizona.edu/support/account/password.html

for details.


math.arizona.edu/support/account/password.html

Basics Connection

From Windows with PuTTY

Open PuTTY.

When ConfigurationWindow opens, change:

Host NamePort

Set to SSH

and open theconnection


Basics Connection

From Cygwin/Apple/Linux with aTerminal

Open a terminal.

Connect with the command

ssh -p PORT USER@SERVER

USER Your username. Mine is dlove

SERVER The server to connect to, e.g.,gila.math.arizona.edu

PORT Port number to connect to. Math department uses31415. Other servers may use the default port, 22,which needs no -p tag.

To connect to gila, I use

ssh -p 31415 [email protected]


Basics Connection

Moving Between Computers

Once connected, move between computers with

ssh SERVER

If USER is omitted, ssh assumes username is same as currentaccount.

To log onto chivo1 from gila (or any Linux computer in thelabs):

ssh chivo1

It will ask for password. Later we’ll see how to avoid that.


Basics Connection

Create a Virtual Desktop

Log onto iguana.math.arizona.edu.

I run code (in file makevnc)1 killall Xvnc42 vncserver -geometry 1136x710 -name ”MathVNC (dlove)”

Geometry customized to fit my laptop screen.

Note the number assigned to your desktop, e.g., “:7” or “:13”

Log on with

vncviewer iguana.math.arizona.edu:13

Or whatever your desktop number is.


Basics Basic Commands

The Structure of a Linux Command

Linux Command Structore

commandname [OPTIONS] ARGUMENTS

The basic command:

1 Begins with the name of the command.

2 Might include some options. As the name suggests, options aregenerally optional.

1 In Linux’s documentation, optional things are in brackets.2 Most options are denoted with a dash and a letter, -r.3 They can be strung together, -rfp. Some require arguments.4 Word length options are generally denoted with two dashes,

--recursive.

3 May include one or more arguments.






The basic command:



1 In Linux’s documentation, optional things are in brackets.2 Most options are denoted with a dash and a letter, -r.

3 They can be strung together, -rfp. Some require arguments.4 Word length options are generally denoted with two dashes,

--recursive.







The basic command:



1 In Linux’s documentation, optional things are in brackets.2 Most options are denoted with a dash and a letter, -r.3 They can be strung together, -rfp. Some require arguments.

4 Word length options are generally denoted with two dashes,--recursive.







The basic command:



1 In Linux’s documentation, optional things are in brackets.2 Most options are denoted with a dash and a letter, -r.3 They can be strung together, -rfp. Some require arguments.4 Word length options are generally denoted with two dashes,

--recursive.




Basic Unix Commands

pwd - print the name of your current/working directory.

cd [directoryname] - change your working directory.

ls [directoryname] - list the contents of directories.

mkdir directoryname - make a new directory.

rmdir directoryname - remove/delete a directory.

cp source destination - copy files and directories.

mv source destination - move files and directories.

rm filenamelist - remove/delete files.

cat filename - concatenate file, i.e., display the contents.

yppasswd - to change your network account password.

logout - log out of a terminal.



Examples & Useful Options

If I log into gila, I can type some commands:

[ d love@gi la ˜ ] $ pwd/u1/ dlove

[ d love@gi la ˜ ] $ l sbin / P i c tu r e s /pub ht tp in t e rne t / Desktop/Publ ic / Documents/makevnc∗ Templates/Music/ Videos /

pwd told me the folder I was in.

ls gave a list of the files and subdirectories in /u1/dlove



More Examples

[ d love@gi la ˜ ] $ l sbin / P i c tu r e s /pub ht tp in t e rne t / Desktop/Publ ic / Documents/makevnc∗ Templates/Music/ Videos /

The / at the end indicates a subdirectory.

The * indicates an executable file.

Notice the prompt, showing:

Username: dloveMachine name: gilaCurrent directory: ~



Useful Options & Other Details

cp source file destination

-r Recursive, copy directory and all contents.-i Interactive, ask before copying over another

file.-f Force, override interactive.-u Update, only copy when source is newer than

destination.

Example cp file newfile

Copy file to newfile.

Example cp file new directory/

Copy file the the subdirectory new directory/,keeping the same name.



Usefule Options & Other Details

ls [directory or files]

-a Dislpay all files, even hidden files.-l Display long form with more information.-R Recursive, display contents of subdirectories

as well.

rm [file]

-r Recursive, remove files from subdirectories.-i Interactive, confirm before deleting.-f Force, override interactive.



Useful Linux Shortcuts

Tab Completion

Pressing the Tab key will complete a keyword or filename inUNIX/Linux. If the completion is not unique, pressing Tab twice willgive a list of possible completions.


Basics Filesystem

Files in Linux

Filenames in Linux can contain any character except / and thenull character.

You should avoid special characters like $, ?, *, \.When using files with special characters or spaces in commands,they need to be escaped with a \.For example, to remove a file named file with spaces*:

rm file\ with\ spaces\*

Files don’t need extensions, but some like .c, .tex are standard.

Files can have multiple extensions, like hw.tex.backup.

Filenames beginning with a . are “hidden,” and not displayed byls without using the -a option.


Basics Filesystem

The UNIX Filesystem

The UNIX/Linux filesystem is quite different than Windows.Some things that might be helpful as a Linux user:

/ The root directory, contains everything else in thefilesystem.

/bin bin is a subdirectory of /, and contains some of themost basic executable files.

/usr Contains most other executables, libraries, anddocumentation files.

/etc “Editable Text Configuration files.” Configurationfiles for the entire system. Can provide examples foryour own config. files.

/tmp Space for temporary files./home Contains home directories for all users.


Basics Filesystem

Math Department Filesystem

The department computers have some nonstandard directoriesunder /:

/scratch Space to put temporary files generated by anycomputations you do.

Computer specific.

/u1,/u2,...,/u6 Home directories on the department computersare located in one of these directories, instead of/home. My directory is in /u1. Yours might besomewhere different.

Common to all department computers (exceptpassword).


Basics Filesystem

File System Abbreviations

UNIX and Linux have abbreviations for some directories:

~ Home directory.

. Current directory.

.. Parent directory.

If I’m in my home directory ~, or /u1/dlove, then:

. is /u1/dlove, and

.. is /u1.


Commands

Wildcards

Linux uses some characters to indicate patterns of text.

? Fill in any single character.

ls hw?.tex lists all filenames like hw1.tex, hwQ.tex, etc.

* Fill in any number of characters

ls hw* lists all files starting with hw, like hw1.tex,hw583.pdf, etc.

cat hw*.tex concatenates all files that start with hw andend with .tex

[] Specify a range for a single character

cp hw[1-3].tex .. Copy files hw1.tex, hw2.tex,hw3.tex to the parent directory.


Commands

More Wildcards

Of course, wildcards can be combined.

rm -f hw[1-4]?.[a,p]* removes, without confirmation, filesthat:

Begin with hw,Followed by a digit between 1 and 4,Followed by another single character, andHave an extension that begins with an a or p


Commands

Output Redirection

Linux prints to the screen a lot, and lets you work with that.

> Write or overwrite output to a file.

cat hw*.tex > allhw.tex Concatenate all files hw*.tex

into allhw.tex.

>> Append output to end of a file.

cat newhw*.tex >> allhw.tex Concatenate the files,add to the end of allhw.tex.

| “Pipe” output into a new command. Example:

tee FILE prints output both to FILE and the screen.COMMAND | tee FILE prints the output of COMMANDto both FILE and the screen.


Commands

man pages

UNIX & Linux systems have a built-in manual: the man pages.

The structure of a man page:

Name A short description of the command.Synopsis The basic form of the command:

Optional arguments enclosed in “[]”.Options with finite choices enclosed in “{}”.

Description A longer description of the command, including alloptions.

Other things in the man page might be:

Examples,Bug reporting,List of similar commands.


Commands Useful List

screen

screen creates virtual terminals that:

Can continue to run when you log out.Allow you to come back whenever you want.

Typing screen creates a new terminal.

Â-d (CTRL-a then d) detaches (leaves) a session.

The -r option (screen -r) reattaches (continues) a session.

Other commands:

Â-c Create another terminal window.Â-n Cycle to next terminal window.Â-p Cycle to previous terminal window.

logout Close the terminal window.



grep

grep PATTERN [FILE...] prints lines in a file that containPATTERN.

grep for prog.c prints all lines in the file prog.c containing thepattern for.

Some options:

-n Print line number.-H Print file name.-i Ignore case.-v Print non-matching lines.-w Only match full words.

Example: grep -nHiw rand01 * in a code I used.



diff

diff FILE1 FILE2 intelligently compares files line by line andprints the differences.

If given directories, it compares files of the same name in each.

Output:

< Indicates a line in the first file.> Indicates a line in the second file.= Indicates common lines.

Options:

-i Ignore case.-w Ignore white-space.-q Print only whether files differ.

Programs kompare and gvimdiff are editors that show differencesbetween files. kompare is easier if you don’t know vim.



Using diff Recursively

The situation: you’ve been given a big code, and you’ve madesome changes.

After some time away, you discover an error in the modified codethat’s not in the original.

But you don’t remember which files you changed!

diff, with the recursive (-r), quiet (-q) and ignore white space(-w) will give a list of files that have changed.

Old code in directory rd/.New code in directory rd.new/.Command: diff -rqw rd rd.new.



quota, ps and top

quota Displays how much space your account is allotted. Use -s

option.

ps Lists running processes.

Without arguments, it shows processes running fromthe current terminal.Use -u to specify a user name, e.g., ps -u dlove.

top Real time list of processes using system resources.

Quit by typing q.Type u to enter a username.



kill & killall

Closing programs from the terminal:

^C Kills the program currently running in the terminal.kill PID Kills process with id number PID

-9 Signal that can’t be blocked.

killall NAME Kill a process by name.

-9 Signal that can’t be blocked.



tar & zip

tar, short for “tape archive” creates archives.

Options:

-c,-r,-x Create, append to, extract archive.-j,-z Compress with bzip2, gzip.

-v Verbose.-f Specify filename for archive

Examples:

tar -cvjf archive.tar.bz2 folder Compress folder intoarchive archive.tar.bz2.

zip and unzip work with .zip archives.



find

find recursively searches through directories to find files withgiven characteristics.

find / -name passwd will find all files named “passwd” on thecomputer.

find -iname resume.tex will find all files named”resume.tex,” but is case insensitive.

find -name "*.sh" -executable will find all executable fileswith name *.sh.

When using wildcards, enclose in quotes to prevent the shell fromexpanding them.

find has a lot of possible tests and options. Seehttp://www.thegeekstuff.com/2009/03/

15-practical-linux-find-command-examples/ for a tutorial ofsome useful ones.


http://www.thegeekstuff.com/2009/03/15-practical-linux-find-command-examples/

http://www.thegeekstuff.com/2009/03/15-practical-linux-find-command-examples/


locate

locate also finds files based on a name or path pattern.

It searches through a database–much faster than find!

But database is only updated periodically.

Many fewer options than find.

locate [OPTIONS] PATTERN is the basic command.



xargs

xargs puts the output from one command into arguments foranother command.

Very useful with find!

To deal with filenames that contain spaces or other difficultcharacters, use print0 (zero) option with find, and -0 (zero)option with xargs.

Command: find OPTIONS -print0 | xargs -0 -I fileCOMMAND file OTHER ARGUMENTS.

-I option replaces file with the output of find.

Examples

find . -name "*.cpp" -print0 | xargs -0I file rm file

find . -name "*.tex" -print0 | xargs -0I file mv file

file.backup


Commands File Transfer

Secure Shell Copy

Basic method is to use scp

Open a terminal.

Connect with the command

scp -P PORT SOURCE DESTINATION

SOURCE Where the file currently is.DESTINATION Where you want to file to be.

PORT Port number to connect to. Math department uses31415. Other servers may use the default port (no -P

tag).



scp Continued

Suppose I want to copy ∼/file.txt from gila to Documents on mycomputer:

scp -P 31415 [email protected]:∼/file.txt∼/Documents/



rsync: File Synchronization

rsync is a much more powerful tool:

Checks files on local machine and server, only transfers files thathave changed.

Only transfers differences between changed files–Very Fast!

Suppose I want to copy ∼/file.txt from gila to Documents on mycomputer:

rsync --rsh=’ssh -p 31415’

[email protected]:∼/file.txt ∼/Documents/

Servers using the default port don’t need the --rsh=’ssh -p 31415’

information.


Commands Executables

Permissions and chmod

Three types of user can have permissions for each file:

user You.group Various subsets of the logins. On math systems, only

group is “users.”others Everyone else.

And three types of permissions:

read Can read files.write Can write or change files.

execute Can run executables or list contents of directories.



Changing Permissions

Permissions are changed with the chmod command.

chmod MODE FILE

Most intuitive use of MODE is:

TYPE±PERM Adds/removes permissions PERM to users of TYPE.TYPE=PERM Gives TYPE exactly permissions PERM

Examples:

chmod u+x file Allow file to be executed.chmod a+X file Allows anyone to execute if file is a directory

or if someone else has execute permission.chmod go-rwx file No one from groups or others can read, write

or execute file.



Running Code on Linux

To run a program, must specify where the program is.

Run program cep1, which is in /home/dlove:/home/dlove/cep1

Can use shortcuts.

Run program nv, in the current directory, type:./nv


Scripting

Bash

When you’re using the command line, you’re actually using a shell.

The shell provides the user interface, and can function like aprogramming language.

Back in the day, some guy named Bourne wrote the Bourne shell,with an executable named sh.

You’ll still see shell scripts with extension .sh.

Then other people developed other shells.

In 1989, some guy named Fox wanted to update the Bourne shell.

Logically, he named it:

The Bourne-Again Shell (Bash).

Bash the default shell in most UNIX/Linux systems.


Scripting

Files in Your Home Directory

Many “hidden” files, beginning with a “.” are in your homedirectory.

Can see them with ls -a option.

Many programs use these files to store configuration information.

Hidden files can almost always be edited with only a text editor.

The most important hidden files:

.bashrc Contains the setup information for the shell..profile Similar to .bashrc.

There are often basic files in /etc.


Scripting

Introduction to Variables

Bash uses variables to store important information.

Variables can be used like in a programming language.

By convention, variables in the shell are named with capital letters.

Variables value is accessed with $.

For example, the PATH variable stores the folders that includeexecutables.

Using the echo command:

[ d love@gi la ˜ ] $ echo $PATH/usr / l o c a l / sb in : / usr / l o c a l / bin : / usr / sb in :/ usr / bin : / sb in : / bin : / usr /games

Can add more by PATH=$PATH:DIRECTORY. E.g., in .bashrc.


Scripting

Scripts

Bash can function as a high-level language useful for many tasks.

Like other languages, Bash has:

For loops,If tests,Changing values of variables.

I have several example codes that I’ve used in my research.


Scripting

An Example Script

1 #! /bin/bash

2 # $1=problem, $2=m

3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then

7 RUN="./rd $1 -s $2 -g $G | tee $1 m$2 $G.dat | grep Rep"

8 echo ${RUN}9 eval ${RUN}10 fi

11 done

12 ./parser $1 m$2 *.dat -o total $1 m$2.dat

13 echo "Done!"


Scripting

An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Program to use to interpret


Scripting

An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

A comment. Variables named as numbers given command linearguments. 1 is the first argument, accessed by $1, etc.


Scripting

An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Basic structure of a for loop


Scripting

An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Basic structure of an if test. -ge means “≥“


Scripting

An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Load a string containing the command into RUN. Variables areevaluated inside quotes.


Scripting

An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Print the string to the screen.


Scripting

An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Execute the code using eval.


Scripting

An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"

Execute another program to consolidate all the output files.


Scripting

An Example Script

1 #! /bin/bash


3 for G in 0100 0200 1000

4 do

5 if [ $2 -ge $gamma ];

6 then



11 done


13 echo "Done!"


Scripting

My Solution to File Transfer

I have three rsync-based scripts to transfer files from mycomputers to gila.

I make sure that the ∼/Documents folder is identical on mydesktop, laptop, and on gila:

ups (“Upstream Backup”) Copies ∼/Documents from mycomputer to gila. If it finds files on gila that are not onmy computer, it deletes them.

dow (“Downstream Backup”) Copies ∼/Documents from gila

to my computer. If it finds files on my copy of∼/Documents that are not on gila, it deletes them.

ms (“Math Sync”) Synchronizes in both direction, but willnot delete files anywhere.


Scripting

My Code

My code has 4 lines that need to be edited:

Change the user name.

MATHSERVER="[email protected]"

The folder on your computer to sync.

LOCALDOCS=∼/DocumentsThe folder on gila to sync.

MATHDOCS="∼/Documents"The options. The ’n’ tells rsync not to change any files. Removethis when you’re confident the code is correct.

OPTS="-vrtun"


Scripting

Check chivos’ Status

Code ccheck (Chivo Check) tells you how busy the chivos are.

1 #!/bin/bash

2 for x in {1..4}3 do

4 echo "chivo${x}"5 ssh chivo${x} ’uptime’

6 done

What it does:

Log onto each chivo, run command uptime and log out.

Look at the load averages to see how much work each computer isdoing.


Scripting

Example Output

chivo1

16:49:55 up 143 days, 1:00, 1 users, load average:

0.67, 0.50, 0.44

chivo2


1.03, 1.05, 1.02

chivo3

16:49:56 up 493 days, 3:41, 1 user, load average:

1.00, 1.00, 1.00

chivo4


3.00, 3.00, 3.00

Use the lowest load average (chivo1) to run your code.


Improvements

Aliases

Aliases make complicated commands simple.

Put them in .bashrc or .profile.

Format is alias NEWCOMMAND=OLDCOMMAND.

For example, the file /etc/skel/.bashrc contains:

alias ll=’ls -alF’

alias la=’ls -A’

alias l=’ls -CF’

alias grep=’grep --color=auto’


Improvements

Networking Aliases

I have some aliases set up specifically for networking:

alias math=’ssh -p 31415 [email protected]’

alias mathi=’ssh -p 31415

[email protected]’

alias mathvnc=’vncviewer iguana.math.arizona.edu:4

-passwd $HOME/.vnc/passwd’


Improvements

Passwordless Connection & Privacy

Most connection methods allow you to connect without passwords.

Make sure that only you have permission to read the required files!


Improvements

SSH Without Password (from homecomputer)

To login from a terminal without giving a password

Run ssh-keygen -t rsa from your own computer.

Accept the default location for the key.

No need to give a passkey.

Make a .ssh folder on the server (e.g., gila):

ssh -p 31415 [email protected] mkdir -p .ssh

Append id rsa.pub to your list of authorized keys on gila

cat .ssh/id rsa.pub | ssh [email protected]

’cat >> .ssh/authorized keys2’


Improvements

SSH Without Password (between mathcomputers)

To move between computers on the Math network

Run ssh-keygen -t rsa on gila.

Accept the default location for the key.

Accept giving no passkey.

Append id rsa.pub to your list of authorized keys on gila

cat .ssh/id rsa.pub >> .ssh/authorized keys2

Script pass free ssh.sh automates this. Available on SWIGwebsite.


Improvements

VNC Without Password

To use VNC without a password, you need only modify your owncomputer.

Run vncpasswd

Accept the default location for the password file.

Enter your password.

There’s no need for a view-only password.

Call your VNC viewer with the -passwd option, and the locationof your password, e.g.,

vncviewer iguana.math.arizona.edu:12 -passwd

$HOME/.vnc/passwd

(Note: $HOME is the same as ~.)


Resources

Linux Documentation Project http://tldp.org/guides.html

Introduction to Linux, a Hands On Guidehttp://tldp.org/LDP/intro-linux/html/index.html

SSH Login Without Passwordhttp://linuxproblem.org/art_9.html

VNC Without Password http://www.dotkam.com/2009/03/22/

vnc-into-remote-server-without-typing-a-password/

http://tldp.org/guides.html

http://tldp.org/LDP/intro-linux/html/index.html

http://linuxproblem.org/art_9.html

http://www.dotkam.com/2009/03/22/vnc-into-remote-server-without-typing-a-password/

http://www.dotkam.com/2009/03/22/vnc-into-remote-server-without-typing-a-password/

The End

Introduction to Linux and the Math Department...

Documents

Transcript of Introduction to Linux and the Math Department...