Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and...
Transcript of Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and...
![Page 1: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/1.jpg)
Introduction to Unix
University of Massachusetts Medical School
February, 2019
![Page 2: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/2.jpg)
![Page 3: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/3.jpg)
DISCLAIMER
For the sake of clarity, the concepts mentioned in these slideshave been simplified significantly.
Most of the command line tools and programs we explain havemany more features that can’t be found here. Unix commandline tools are very well documented and their manual pages arefreely available.
These slides are prepared for the users of MGHPCC.MGHPCC system has features that don’t exist in standard Unixsystems such as loading modules.
A rigorous treatment of topics on Unix and Bash can be foundin various books in several levels.
![Page 4: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/4.jpg)
Some References
Unix:Unix in a Nutshell , Arnold Robbins, ISBN-13: 978-0596100292Learning the bash Shell: Unix Shell Programming, Cameron Newham, ISBN-13:978-0596009656
Umass High Perfotmance Computing:http://wiki.umassrc.org/
Bioinformatics:UCSC genome bioinformatics: https://genome.ucsc.edu/Sam file format: samtools.github.io/hts-specs/SAMv1.pdfSamtools: http://samtools.sourceforge.net/Bedtools: http://bedtools.readthedocs.org/en/latest/bowtie2: http://bowtie-bio.sourceforge.net/bowtie2/manual.shtmltophat: http://ccb.jhu.edu/software/tophat/index.shtml
4 / 161
![Page 5: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/5.jpg)
Basic Components of a Computer
Input devices: Keyboard, mouse, scanner, terminal ,...Output devices: Screen, speaker, printer, terminal ,...Processors: CPU (e.g., intel i7), GPU (GeForce), ...Memory: RAM, Level 3 Cache, ...Storage Devices: Flash Drives, Magnetic Drives, OpticalDisks,...
Data and software are stored in the storage devices via inputdevices. Software and data are loaded into the memory andprocessed by the processors and we get the results via theoutput devices.
5 / 161
![Page 6: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/6.jpg)
Computer Architecture
CPU (Central Processing Unit)
Registers
Arithmetic Logic Unit
Control Unit
Memory
(address, contents)
Data
Address
(101...1, 010....1)
(111...1, 110....1)
(100...1, 000....1)
This is called the Von Neumann architecture.
A cpu works in
Fetch
DecodeExecute cycle to carry outinstructions.
Consequently, a processor of clock rate 4.5 GHz performs, ingeneral, better than a processor (in the same family) of clockrate 4 GHz.
6 / 161
![Page 7: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/7.jpg)
Fetch-Decode-Execute
Fetch: CPU has a special register that holds the memorylocation of the next instruction. It sets the address busaccordingly and gets the the next instruction.
Decode: Once it gets the instruction, it can figure out thememory location of the next instruction. It understands theinstruction.
Execute: It performs the instruction. ‘
7 / 161
![Page 8: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/8.jpg)
Note
In today’s computers, many improvements have been made tooptimize the cost and the performance.
Examples: Multiple cores, several levels of caches on the CPU,pipelining, ...
8 / 161
![Page 9: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/9.jpg)
Wall Clock Time and CPU Time
We can talk about two different time measures when we run aprocess.
Wall Clock Time: The time difference between the start andend of the execution of a process.
CPU Clock Time: The actual time CPU spent in thefetch-decode-execute cycle to run the process.
9 / 161
![Page 10: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/10.jpg)
Example
A simple experiment can show us that, even two differentprocesses take almost the same time to finish, their CPU timeusage can be different: First, create a dummy file of 1 GB.
$ dd if=/dev/zero of=dummy.img bs=1024 count=1024000
Then compute the hash sum of this file and time it
$ time md5sum dummy.img
Now run sleep for the same amount of seconds
$ time sleep 10
Notice the difference in user + sys between two runs.
10 / 161
![Page 11: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/11.jpg)
Programming
CPU (Central Processing Unit)
Registers
Arithmetic Logic Unit
Control Unit
Memory
(address, contents)
Data
Address
(101...1, 010....1)
(111...1, 110....1)
(100...1, 000....1)
From a technical point of view, we can define programing asgenerating the memory content in such a way that when theCPU processes the contents of the memory, we’ll achieve agiven objective.
11 / 161
![Page 12: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/12.jpg)
Programming Languages
Today, programing is very rarely done by manually generatingthe contents of the memory.We use programming languages instead (most of the time).
Why?
Consider:Binary Code:1011000011.....1101
R Code:for (gene_expression in RNA_Seq_Data){print(mean(gene_expression))}
12 / 161
![Page 13: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/13.jpg)
Programming Languages
Today, programing is very rarely done by manually generatingthe contents of the memory.We use programming languages instead (most of the time).
Why?
Consider:Binary Code:1011000011.....1101
R Code:for (gene_expression in RNA_Seq_Data){print(mean(gene_expression))}
13 / 161
![Page 14: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/14.jpg)
Programming Languages
Today, programing is very rarely done by manually generatingthe contents of the memory.We use programming languages instead (most of the time).
Why?
Consider:Binary Code:1011000011.....1101
R Code:for (gene_expression in RNA_Seq_Data){print(mean(gene_expression))}
14 / 161
![Page 15: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/15.jpg)
Programming Languages
A programming language is a formal definition of a system thattranslates human instructions to Binary (Machine) code. Thistranslation is done by a piece of software which can be acompiler or an interpreter.
for (gene_expression in RNA_Seq_Data){print(mean(gene_expression))} interpreter eventually executes−−−−−−−−−−−−−−−−−−−−−−−−→
0101...011111
15 / 161
![Page 16: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/16.jpg)
Processes & Threads
A process can be defined as the running instance of somesoftware where the instructions are sequentially executed bythe CPU.
A thread is the smallest unit of a process that can be runindependently.
Both threads and processes can run in parallel. The maindifference between them is that different threads of a processcan share the same system resources (memory space, filesand etc) whereas processes can not.
16 / 161
![Page 17: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/17.jpg)
Processes & Threads
CPU (Central Processing Unit)
Registers
Arithmetic Logic Unit
Control Unit
Memory
(address, contents)
Data
Address
(101...1, 010....1)
(111...1, 110....1)
(100...1, 000....1)
The Von Neumann architecture allows one process at a time torun. So how can multiple processes / threads can run inparallel?
17 / 161
![Page 18: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/18.jpg)
Processes & Threads
Core N
CPU (Central Processing Unit)
Registers
Arithmetic Logic Unit
Control UnitMemory
(address, contents)
Data
Address
(101...1, 010....1)
(111...1, 110....1)
(100...1, 000....1)
Registers
Arithmetic Logic Unit
Control Unit
Core 1
.............................................
.............................................
.............................................
Modern processors have more than one core.So, if multi-cores are available and your software can takeadvantage of this, the run time of your process will be shorter.It is advantageous to take this into consideration whensubmitting jobs and setting multithreading parameters of yourbioinformatics tools. Examples to come in the following classes.
18 / 161
![Page 19: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/19.jpg)
Exercise
You are using an short-read aligner software. Assume that itsspeed is linearly proportional to the number of threads it isworking with. You have run this software using 3 threads and ittook 90 minutes to align your RNA-Seq data. How much time,roughly, do you expect to wait if you run the same software, onthe same data and, on the same computing environment, using5 threads?
90× 35= 54 minutes
If we run it using a single thread, it works 3 times slower ,compared to 3 threads, so the expected running time is90× 3 = 270 minutes. If we use 5 threads, it will be 5 timesfaster, compared to single thread, so 270 / 5 = 54 minutes.
19 / 161
![Page 20: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/20.jpg)
Exercise
You are using an short-read aligner software. Assume that itsspeed is linearly proportional to the number of threads it isworking with. You have run this software using 3 threads and ittook 90 minutes to align your RNA-Seq data. How much time,roughly, do you expect to wait if you run the same software, onthe same data and, on the same computing environment, using5 threads?
90× 35= 54 minutes
If we run it using a single thread, it works 3 times slower ,compared to 3 threads, so the expected running time is90× 3 = 270 minutes. If we use 5 threads, it will be 5 timesfaster, compared to single thread, so 270 / 5 = 54 minutes.
20 / 161
![Page 21: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/21.jpg)
Process Management
What if you want to run 40 processes simultaneously usinga 4 core CPU? (Process management)How can you make sure that a process doesn’t read / writethe memory contents of another process? (Memorymanagement)What if you want to make a single computing environmentavailable to many users and make sure that systemresources are safely and fairly shared? (File permissionsand etc.)
This is orchestrated by a group of software called OperatingSystem.
21 / 161
![Page 22: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/22.jpg)
What is an Operating System?
An operating system is a collection of software that managessystem resources (such as memory, cpu(s), display and etc.)and provide applications a simpler interface to the systemhardware.
22 / 161
![Page 23: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/23.jpg)
Operating System
Application
Operating System
Hardware
CPU Memory Hard Disk .....
23 / 161
![Page 24: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/24.jpg)
Unix
Developed inthe 1960’s byKen Thompson,Dennis Ritchie.It became verypopular in the70’s and 80’s.
24 / 161
![Page 25: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/25.jpg)
"Contrary to popular belief, Unix is user friendly. It just happensto be very selective about who it decides to make friends with."–anonymous
25 / 161
![Page 26: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/26.jpg)
Unix-like operating systems
Unix-like operating systems "behave" like the original Unixoperating system and comply (at least partially) with POSIX(portable operating system interface) standards.
Examples: Linux, OS X, Free BSD, Solaris, Hurd.
Unix ≡ Unix-like ≡ Unix-clone
26 / 161
![Page 27: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/27.jpg)
What is a Terminal?
A terminal is a software that emulates a teletype writer terminalused in the early days of Unix.
27 / 161
![Page 28: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/28.jpg)
Unix Shell
A shell is a software that runs insidea terminal and interprets and executesuser commands.
One of the most popular shells be-ing used today is called BASH (BourneAgain Shell). It is also the default shell inthe UMass HPC cluster, OS X and manyLinux distributions.
User
Shell
Kernel
28 / 161
![Page 29: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/29.jpg)
Convention
$ denotes the Bash command line prompt. It is not meant to betyped. All Bash commands will appear in a gray box.
$ bashcommand
29 / 161
![Page 30: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/30.jpg)
Hello Bash
We connect to the MGHPCC server.
$ ssh [email protected]
Can’t see your password being typed on the screen?
That is a good sign!For security reasons, you shouldn’t see your password on thescreen when you type it!
30 / 161
![Page 31: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/31.jpg)
Hello Bash
We connect to the MGHPCC server.
$ ssh [email protected]
Can’t see your password being typed on the screen?
That is a good sign!For security reasons, you shouldn’t see your password on thescreen when you type it!
31 / 161
![Page 32: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/32.jpg)
Hello Bash
Let’s verify that we are at the right place.
$ hostname
To print something on the screen, we use echo
$ echo Hello World
32 / 161
![Page 33: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/33.jpg)
Navigating the Unix File System
/
bin etc mediahome
user1
...
user2 ...
script.sh Documents
33 / 161
![Page 34: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/34.jpg)
Exploring the Directories
Bash always has a working directory. To see the workingdirectory,
$ pwd
To see the contents of the working directory,
$ ls
To see the contents of a particular directory, provide the path ofthe directory.
$ ls /bin
displays th contents of the directory /bin
34 / 161
![Page 35: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/35.jpg)
Exploring the Directories
Bash always has a working directory. To see the workingdirectory,
$ pwd
To see the contents of the working directory,
$ ls
To see the contents of a particular directory, provide the path ofthe directory.
$ ls /bin
displays th contents of the directory /bin
35 / 161
![Page 36: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/36.jpg)
Exploring the Directories
Bash always has a working directory. To see the workingdirectory,
$ pwd
To see the contents of the working directory,
$ ls
To see the contents of a particular directory, provide the path ofthe directory.
$ ls /bin
displays th contents of the directory /bin
36 / 161
![Page 37: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/37.jpg)
Command Line Arguments
A program’s execution and / or output format can be modified,according to our needs, by providing command line arguments.
$ ls︸︷︷︸executable
-lh︸ ︷︷ ︸argument 1
/bin︸ ︷︷ ︸argument 2
37 / 161
![Page 38: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/38.jpg)
More on ls
We can get more information about the files or directories byrunning ls with additional arguments.
Compare
$ ls
with
$ ls -l
38 / 161
![Page 39: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/39.jpg)
A closer look at ls
ls lists the given directory contents.To see the complete list of options,
$ man ls
This will display the manual page of ls.
Some commonly used options are:-l : list contents in more detail-A : list all files in the direcory-h : when used with the -l option,
prints file sizes in KB, MB, GB, and etc.to reduce the number of digits on the screen.
These options can be grouped
$ ls -lh
39 / 161
![Page 40: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/40.jpg)
A closer look at ls
ls lists the given directory contents.To see the complete list of options,
$ man ls
This will display the manual page of ls.
Some commonly used options are:-l : list contents in more detail-A : list all files in the direcory-h : when used with the -l option,
prints file sizes in KB, MB, GB, and etc.to reduce the number of digits on the screen.
These options can be grouped
$ ls -lh
40 / 161
![Page 41: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/41.jpg)
Let’s first have some files and folders to experiment with.
$ /tmp/bootcamp.pl
Upon success, you will see/home/user_name/bootcamp
41 / 161
![Page 42: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/42.jpg)
Changing the working directory
First, we see the working directory
$ pwd
Now, lets change the directory to the bootcamp directory.
$ cd ~/bootcamp
We can verify the change by
$ pwd
42 / 161
![Page 43: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/43.jpg)
Tab for Autocompletion
You can use the tab key for autocompletion. This can save youtime and typing for long file / directory names.
$ touch this_is_a_somewhat_long_name.txt
To delete this file, after rm, type this_ and hit the tab key.
$ rm this_
43 / 161
![Page 44: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/44.jpg)
Creating Directories
$ mkdir directory_path
Note that
$ mkdir /home/username/bootcamp/my_new_directory
is the same as
$ mkdir ~/bootcamp/my_new_directory
44 / 161
![Page 45: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/45.jpg)
Copying Files
$ cp sourcefile(s) destination_path
Let’s copy our sample files into the home folder.
$ cp ~/bootcamp/transcriptomics.tar.gz ~/transcriptomics.tar.gz
45 / 161
![Page 46: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/46.jpg)
Copying Directories
$ cp source_directory destination_directory
will not work in general.You need to provide the -r parameter.
$ cp -r source_directory destination_directory
Another useful option is -v which prints the files being copied.
$ cp -rv source_directory destination_directory
46 / 161
![Page 47: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/47.jpg)
Moving Files
$ mv source_file destination_file
47 / 161
![Page 48: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/48.jpg)
Deleting Files
rm : Remove comand
$ rm file_path
To remove non-empty directories, use rm with the -r option.
$ rm -r directory
48 / 161
![Page 49: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/49.jpg)
rm is a dangerous command
There is no undoing of rm!There is no command like unrm my_file orbring_back my_file.A file is a bunch of 0’s and 1’s in a storage device and thelocation of these 0’s and 1’s are stored in a table. Once youremove the file, it is very likely that those 0’s and 1’s will beoverwritten and hence flipped. So it is very unlikely that you’llrecover the file.
You may want to use it in the interactive mode not to delete filesaccidentally.
$ rm -i file
49 / 161
![Page 50: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/50.jpg)
Back up Your Data!
It is good practice to backup your data regularly. You can use aremote machine or an external drive to take backups.
50 / 161
![Page 51: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/51.jpg)
File Path
~ : Home directory
. : current directory
.. : parent directory
If a file path begins with /, then it is an absolute path.It doesn’t depend on the current working directory.
If a file path begins WITHOUT /, then it is a relativepath. The path is defined according to the currentdirectory. So, the path depends on the currentworking directory.
51 / 161
![Page 52: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/52.jpg)
Examples: Absolute vs Relative Path
/tmp/bootcamp/bootcamp.pl : Begins with / so,absolute path.
bootcamp.pl : This is the botcamp.pl file in the workingdirectory. So, relative path
../bootcamp.pl : This is the botcamp.pl file in the parentdirectory of the working directory. So, relative path
~/bootcamp.pl : This is the file in the home directory of thecurrent user. So, relative path.
52 / 161
![Page 53: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/53.jpg)
Files
Technically, every file is a binary file.
Yet, it is common practice to group the files into two:
Text Files : Contains only printable characters.Binary Files : Files that are not text files.
Binary files are generally more compact and therefore smallerin size. They are read and processed by specific applications.
53 / 161
![Page 54: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/54.jpg)
Viewing text files
Text files are of extreme importance in bioinformatics.There are many tools to view and edit text files.less is a very useful program.
$ less text_file
For details read the manual page
$ man less
54 / 161
![Page 55: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/55.jpg)
Editing Text Files
For a very simple text editor:
$ nano
vi and emacs are very powerful text editors with sophisticatedfeatures.
55 / 161
![Page 56: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/56.jpg)
File Permissions
Let’s try to run a bash script.
$ ./hello.sh
Let’s try another one
$ ./other_hello.sh
Why didn’t this work?
56 / 161
![Page 57: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/57.jpg)
File Permissions
Let’s try to spy on Hakan and see what he is up to.
$ ls /home/ho86w
In Unix, users can only access (read, write or execute) files thatthey have permission to.
57 / 161
![Page 58: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/58.jpg)
File Permissions
Let’s try to spy on Hakan and see what he is up to.
$ ls /home/ho86w
In Unix, users can only access (read, write or execute) files thatthey have permission to.
58 / 161
![Page 59: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/59.jpg)
Seeing File Permissions
A typical output of
$ ls -lh
is
-rw-r–r– 1 ho86w moore 11M Nov 5 00:41 RPKM.csv-rwxrwxrwx 1 ho86w moore 143B Oct 27 02:53 my_script.shdrwxr-xr-x 21 ho86w moore 714B Jan 24 13:52 sam_files
59 / 161
![Page 60: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/60.jpg)
r : readw : writex : execute
rwx︸ ︷︷ ︸user
r-x︸︷︷︸group
r- -︸︷︷︸others
User : Has read, write and execute access.
Group : Has read and execute access but can’t write to the file
Others : Has only read access.
60 / 161
![Page 61: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/61.jpg)
Changing File Permissions
chmod: Change file mode bits.
$ chmod filemode file
To give only read access to other users,
$ chmod o=r myscript.sh
To give read and execution access together,
$ chmod o=rx myscript.sh
61 / 161
![Page 62: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/62.jpg)
What is a Process?
A computer process is an instance of a program. It takes upspace in the memory and its instructions are executed in theCPU.
62 / 161
![Page 63: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/63.jpg)
Processes
You can list your running processes
$ ps -u username
where username is your username. You can see them in moredetail with the -F option
$ ps -u username -F
63 / 161
![Page 64: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/64.jpg)
Terminating a process
You can terminate a process by
$ kill -9 processID
where processID is the process id that can be obtained bythe ps command.
64 / 161
![Page 65: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/65.jpg)
Background Processes
To send a process to the background, first put the process onhold by pressing ctrl and z, and then send it to thebackground by
$ bg
You can check your processes by
$ jobs
To put a back to the foreground, first get the process id by
$ jobs
and then
$ fg % processNumber
65 / 161
![Page 66: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/66.jpg)
To start a process on the background, put & at the end of yourcommand
$ command &
66 / 161
![Page 67: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/67.jpg)
Working with Compressed Files
A common type of binary files is zipped files. We can savespace by zipping text files.We usegzip to compress / decompress filestar to pack files into one file for archiving purposes.To compress a file
$ gzip file
To decompress, reads.fastq.gz,
$ gzip -d reads.fastq.gz
67 / 161
![Page 68: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/68.jpg)
tar files
Say we collect our results on a project in one directory. Wewant to send this directory in an email attachment to acollaborator. How do we do that?
/
bin etc mediahome
user1
...
user2 ...
script.sh Documents →
68 / 161
![Page 69: Introduction to UnixUnix Shell A shell is a software that runs inside a terminal and interprets and executes user commands. One of the most popular shells be-ing used today is called](https://reader030.fdocuments.us/reader030/viewer/2022040910/5e83c57e19a4ae5c540c2842/html5/thumbnails/69.jpg)
tar files
To pack a directory
$ tar -cvf arcive.tar directory
To pack a directory in a zipped tar file
$ tar -czvf archive.tar.gz directory
To get our directory back from the tar file
$ tar -xvf archive.tar
To get our directory back from the tar.gz file
$ tar -xvzf archive.tar.gz
69 / 161