“IBS574-ComputationalBiology&Bioinformatics”

Spring2018,Tuesday(01/30),2.00-4.00PM

Linuxshell&shellscripting-I

AshokR.DinasarapuPh.DScientist,Bioinformatics

Dept.ofHumanGenetics,EmoryUniversity,Atlanta

11/30/2018,2-4PM

•  KernelisthecorethatprovidesbasicservicesforallotherpartsoftheOS.

•  ShellistheoutermostpartofanOSthatinteractswithusercommands•  commandlineinterpreter•  sh:bash(bourneagainshell),csh,ksh,dash

•  TerminalemulatorgivesusaccesstotheShell•  xterm,rxvt,genome-terminal•  console=physicalterminal

2

Kernel,Shell&TerminalfrequentlyusedtermsinUNIX-likeoperatingsystems(OS)

1/30/2018,2-4PM

3

Kernel,Shell&TerminalfrequentlyusedtermsinUNIX-likeoperatingsystems(OS)

terminal

1/30/2018,2-4PM

•  Shellpromptwillusuallyinclude

[root@machinename~]# /root

[user_name@machinename~]$/home/user_name

Console/Terminal

4

Linuxfilesystemhierarchy1/30/2018,2-4PM

5

Easiestwaytodothisis…

Starttyping!

1/30/2018,2-4PM

•  SSHallowsyoutoconnecttoyourserversecurelyandperformLinuxcommand-lineoperations.

sshuser_name@blnx1.emory.edu

Alternatively,https://blnx1.emory.edu:22443/

user_name@blnx1:~$

•  ~meansyourhomedirectory(/home/user_name) •  SSHfromWindows:downloadPuTTY

6

ConnectingServerviaSSH

1/30/2018,2-4PM

•  WhatisthedefaultShellforblnx1?Usage:ls–la/bin/*sh /bin/sh->dash

•  Whatisthedefaultterminalemulator?Usage:ls-la/usr/bin/*terminal*

/usr/bin/gnome-terminal

•  Whatisphysicalterminal?

7

Console/Terminal

1/30/2018,2-4PM

Somesimplecommands:•  env (currentenvironment)•  date (fordate)•  cal (forcalendar)•  dforfree (diskspaceormemory)•  exit (closeterminal/session)

Usage:env (thenpressenterkey)

8

Console/Terminal

1/30/2018,2-4PM

NavigationofthefilesystemCommands:pwd,ls,cd

•  pwd (print/currentworkingdir)•  ls (listingdirectoriesandfiles)

ls-l (longlistingformat)ls-la (withhiddenfiles)

Usage:pwd /home/user_name

Usage:ls-l91/30/2018,2-4PM

•  mkdir makeasub-directory

Usage:mkdirdocument

mkdir-pdocument/resumemkdir-pdocument/{resume,cv,address}

•  cdchangedirectory

Usage:cddocument/resume (enterdir)

cd../.. (returntopreviousdirectory) 10

NavigationofthefilesystemCommands:pwd,ls,cd,mkdir

1/30/2018,2-4PM

cd~ (changetohomedir)cd (changetohomedir)

cd/ (changetorootdir)

cddocument (relativepath)cd~/document (absolutepath)cd/home/user_name/document

*with$HOMEenvironmentvariable11

NavigationofthefilesystemCommands:pwd,ls,cd

1/30/2018,2-4PM

Ifadirectorynamehaswhitespace–

mkdir‘MyData’

Usage:cdMy\Data cd"MyData” cd'MyData'

12

NavigationofthefilesystemCommands:pwd,ls,cd,mkdir

Anon-quotedbackslash,\,isusedasanescapecharacterinbash.

1/30/2018,2-4PM

Chooseatexteditor:emacs,Vim

Usage:vihome.txt

INSERTmode:presskeyslikeiORa&starttyping."i"willletyouinserttextjustbeforethecursor."I"insertstextatthebeginningofthecurrentline."a"willletyouinserttextjustafterthecursor,and"A"willletyoutypeattheendofthecurrentline.

Create/Edittextfiles

131/30/2018,2-4PM

Typethefollowingtext:

Create/Edittextfiles

Street:201DowmanDriveCity:AtlantaState:GeorgiaCountry:USAZip:30322

141/30/2018,2-4PM

SAVEmode:pressesckeyAND:q!fornottosaveOR

:xtosaveandexit.

15

HowTo:SavefileinViTextEditor

1/30/2018,2-4PM

Alternatively,use:wqforwriteandquit(saveandexit)

Chooseatexteditor:emacs,Vim

Usage:vicollege.txt

Create/Edittextfiles

16

Street:954GatewoodRdCity:AtlantaState:GeorgiaCountry:USAZip:30329

1/30/2018,2-4PM

Listingfiles,directories&it’scontents

17

•  Listfilesrecursively Usage:ls-R

document/resumedocument/cvdocument/addresshome.txtcollege.txtMyData

1/30/2018,2-4PM

Listingfiles,directories&it’scontents

18

CopyMoveSymlink

•  Listfilesrecursively Usage:ls-R

document/resumedocument/cvdocument/addresshome.txtcollege.txtMyData

1/30/2018,2-4PM

•  Copyafile

Usage:cphome.txtdocument/address/

•  Copyallfiles

Usage:cp*.txtdocuments/address/

“cp”–keepstheoriginalfile&makesaduplicateofit

19

CAREFUL!Ifthedestinationfilealreadyexists,itwillbeoverwrittenwithoutaconfirmationprompt.Use“cp-i”interactiveoptionforaprompt.

1/30/2018,2-4PM

•  Moveafile

Usage:mvhome.txtdocument/cv/

•  Moveallfiles

Usage:mv*.txtdocument/cv/

Renamefiles&directoriesusing“mv”movecommand

20

CAREFUL!Ifthedestinationfilealreadyexists,itwillbeoverwrittenwithoutaconfirmationprompt.Use“mv-n”interactiveoptionforaprompt.

1/30/2018,2-4PM

•  ln:createshardlink,defaultln~/document/cv/home.txt~/document/resume/

•  ln-s:symboliclinkln-s~/document/cv/college.txtdocument/resume/

Symboliclinkisaspecialkindoffilethatpointstoanotherfile

211/30/2018,2-4PM

•  view/read Usage:cathome.txt

•  concatenate&viewUsage:cathome.txtcollege.txt

•  concatenate&redirectionUsage:cathome.txtcollege.txt>add.txt

Thecatcommandisusedfor:

22

WARNING:ifadd.txtalreadyexists,itwillbeover-written.Use>>toappendtheoutputtotheexistingfile.

1/30/2018,2-4PM

•  less(readwithcursor)Usage:lessadd.txt

•  more(readpagebypage)Usage:moreadd.txt

“moreorless”:terminalpagercommandsinUnix-likesystems

23

Theprogramlessormoredoesnotrequirethewholefiletobeloadedinmemorytoviewpartsofit.

1/30/2018,2-4PM

•  Viewthefirst5lines Usage:head–n5add.txt

•  Viewthelast4linesUsage:tail–n4add.txt

head&tailletsyouseethefirst&lastfewlinesofatextfile

24

Bydefault,headortailreturnsthefirstorlast10lines,respectivelyofeachfilethatisprovidedtoit.

1/30/2018,2-4PM

•  sendingdatafromoneprogramtoanother Usage:cathome.txt|less

•  Redirectingtoafile Usage:head-2home.txt>lines.txt

wc-l<home.txt>count.txt

“pipe&redirect”

251/30/2018,2-4PM

Whatdoes“>/dev/null2>&1”mean?

•  ArithmeticinPOSIXshellsisdonewith$anddoubleparentheses:

echo$((2+2)) STDIN (<0):keyboard STDOUT (>1):screen STDERR (>2):screen

echo$((2+2))>result.txt

(STDOUTredirectedintoafile,result.txt)

echo$((2+2))>/dev/null2>&1

(TolistenonlytoSTDERR(2>&1),STDOUTredirectedinto/dev/null&STDERRredirectedtoSTDOUT).

261/30/2018,2-4PM

Whatdoes“>/dev/null2>&1”mean?

•  ArithmeticinPOSIXshellsisdonewith$anddoubleparentheses:

echo$((2+2))>/dev/tty STDIN (<0):keyboard STDOUT (>1):screen STDERR (>2):screen

echo$((2+2))>result.txt

(STDOUTredirectedintoafile,result.txt)

echo$((2+2))>/dev/null2>&1

(TolistenonlytoSTDERR(2>&1),STDOUTredirectedinto/dev/null&STDERRredirectedtoSTDOUT).

27

/dev/ttyisaspecialfile,representingtheterminalforthecurrentprocessUse,ps–a(toseeprocessesattachedtottys)InLinux,theconsoleappearsasseveralterminals(ttys)

1/30/2018,2-4PM

28

Createafile(test_error.sh)withthefollowingtwolinesofcode#!/bin/shecho$((2+'x'))Runas./test_error.sh./test_error.sh2>/dev/tty./test_error.sh2>error_file

1/30/2018,2-4PM

•  Searchafileforkeywords

Usage:grepStateadd.txt

•  ignorecaseoption Usage:grepstateadd.txt grep-istateadd.txt

SearchingaPATTERNwithgrep

29

"globalregularexpressionprint"

Regularexpressionsareusedtosearchandmanipulatethetext,basedonthepatterns[Beginningofline(^);Endoftheline($)]

1/30/2018,2-4PM

1  India2  USA3  Ireland4  UK5  Canada

30

“jointwofiles”

Usage:joincountry.txtcity.txt>file.txt

Usage:vicountry.txt

Usage:vicity.txt

1  NewDelhi2  WashingtonDC3  Dublin4  London5  Toronto

•  Joinstherowsoftwofileswhichshareacommonfieldofdata.

1/30/2018,2-4PM

•  Threemaintypeofpermissions•  Read(r),write(w),andexecute(x)•  3typesofusers(user,group&other)

Usage:ls-l

Changethepermissionsoffiles

311/30/2018,2-4PM

•  Threemaintypeofpermissions•  Read(r),write(w),andexecute(x)•  3typesofusers(user,group&other)

Usage:chmodu=rwx,g=rx,o=rhome.txt

Changethepermissionsoffiles

321/30/2018,2-4PM

•  Threemaintypeofpermissions•  Read(r),write(w),andexecute(x)•  3typesofusers(user,group&other)

Usage:chmodu=rwx,g=rx,o=rhome.txt

Changethepermissionsoffiles

33

Usage:chmod754home.txt

4standsfor"read",2standsfor"write”,1standsfor"execute",&0standsfor"nopermission"

1/30/2018,2-4PM

34

Createafile(test_error.sh)withthefollowingtwolinesofcode#!/bin/shecho$((2+'x'))Runas(chmod+xtest_error.sh)./test_error.sh./test_error.sh2>/dev/tty./test_error.sh2>error_file

1/30/2018,2-4PM

•  rmremovesfilesblindly,withnoconceptof'trash’!!!

•  Removeafile Usage:rmdocument/cv/add.txt

•  RemovedirectoryrecursivelyalongwithallofitscontentsUsage:rm-rdocument

Deletefiles&directories

35

Write-protectedfilesprompttheuserforaconfirmation(withayandann)beforeremoval.Use-f(i.e.,force)toremoveallspecifiedfiles,whetherwrite-protectedornot.

1/30/2018,2-4PM

HowtosetupcommandaliasesinLinux

•  Openyour.bashrcvi~/.bashrc

•  Addthealiasaliasu="cd..;ls”

•  PathadjustmentsexportPATH="$PATH:/<here_new_path>”

•  Reloadthe“.bashrc”file source~/.bashrc

361/30/2018,2-4PM

Listofarchiveand/orcompressionformats

Fileextension Officialname Description.tar Tapearchive Archiving.bz2 bzip2 Compression.gz gzip Compression.tar.gz(.tgz) tarwithgzip both.tar.bz2 tarwithbzip2 both.zip ZIP both

DataarchivinginLinux

371/30/2018,2-4PM

•  Tocompressasinglefile(.gz)

Usage:gziphome.txt

•  Archive&compress

Usage:tar-jcvfhome.tar.bz2home.txt

j:usebzip2compress(z:usegzipcompress)c:compressv:verbosef:file

Data/Filecompression

381/30/2018,2-4PM

wgethttps://github.com/samtools/samtools/releases/download/1.3.1/samtools-1.3.1.tar.bz2

•  Viewthecontentsofatar.bz2file

Usage:tar-jtvfsamtools-1.3.1.tar.bz2

•  ExtractthecontentsUsage:tar-jxvfsamtools-1.3.1.tar.bz2

Data/Filedecompression

39

t:listthecontents;x:extractthecontentsofanarchive

Downloadfileusingwget

1/30/2018,2-4PM

WhatisSAMTOOLS?

40

•  SAMTOOLSisasuiteofprogramstohandlehigh-throughputsequencingdata.

•  UsesSAM(SequenceAlignment/Map)/BAM(BinaryAlignment/Map)files.

•  VariantCallFormat(VCF),mostlikelystoredinacompressedmanner.samtoolsmpileup|\

bcftoolscall>name.vcf1/30/2018,2-4PM

41

•  CompressingVCFfileswithBGZipandindexingitwithTabixisthestandardwayVCFfilesarestored.

•  bgzip–Blockcompression/decompressionutility.

•  tabix–GenericindexerforTAB-delimitedgenomepositionfiles.

•  DownloadBGZip’pedVCFfilefromhttp://vat.gersteinlab.org/datasets.php

VCFfilecompressionwithBGZip&indexingitwithTabix

1/30/2018,2-4PM

VCFfilecompressionwithBGZip&indexingitwithTabix

42

•  View/Readcompressedfilegzip-cdCEU.low_coverage.2010_07.indel.genotypes.vcf.gz|head-20zcatCEU.low_coverage.2010_07.indel.genotypes.vcf.gz|head-20

•  Decompress,gzip(compress)&indexgunzipCEU.low_coverage.2010_07.indel.genotypes.vcf.gzgzipCEU.low_coverage.2010_07.indel.genotypes.vcftabixCEU.low_coverage.2010_07.indel.genotypes.vcf.gzNotaBGZFfile:CEU.low_coverage.2010_07.indel.genotypes.vcf.gztbx_index_buildfailed:CEU.low_coverage.2010_07.indel.genotypes.vcf.gz

•  Decompress,bgzip(compress)&indexgunzipCEU.low_coverage.2010_07.indel.genotypes.vcf.gzbgzipCEU.low_coverage.2010_07.indel.genotypes.vcftabixCEU.low_coverage.2010_07.indel.genotypes.vcf.gzCEU.low_coverage.2010_07.indel.genotypes.vcf.gz.tbi1/30/2018,2-4PM

43

Rsync(RemoteSync)commandforcopyingandsynchronizingfiles&directoriesremotelyaswellaslocallyinLinux/Unixsystems

rsync–av~/script/CorrPlot.Radinasarapu@blnx1.emory.edu:~/adinasarapu@blnx1.emory.edu'spassword:buildingfilelist...doneCorrPlot.Rsent3310bytesreceived42bytes103.14bytes/sectotalsizeis3177speedupis0.95

1/30/2018,2-4PM

44

Thank you

Practice Makes Perfect

1/30/2018,2-4PM