You’ve experienced the shiny, point-and-click surface of your Linux computer—now dive below and explore its depths with the power of the command line.

The Linux Command Line takes you from your very first terminal keystrokes to writing full programs in Bash, the most popular Linux shell. Along the way you’ll learn the timeless skills handed down by generations of gray-bearded, mouse-shunning gurus: file navigation, environment configuration, command chaining, pattern matching with regular expressions, and more.

In addition to that practical knowledge, author William Shotts reveals the philosophy behind these tools and the rich heritage that your desktop Linux machine has inherited from Unix supercomputers of yore.

As you make your way through the book’s short, easily digestible chapters, you’ll learn how to:

• Create and delete files, directories, and symlinks

• Administer your system, including networking, package installation, and process management

B A N I S H Y O U R M O U S E

B A N I S H Y O U R M O U S E

• Use standard input and output, redirection, and pipelines

• Edit files with Vi, the world’s most popular text editor

• Write shell scripts to automate common or boring tasks

• Slice and dice text files with cut, paste, grep, patch, and sed

Once you overcome your initial “shell shock,” you’ll find that the command line is a natural and expressive way to communicate with your computer. Just don’t be surprised if your mouse starts to gather dust.

A B O U T T H E A U T H O R

William E. Shotts, Jr., has been a software professional and avid Linux user for more than 15 years. He has an extensive background in software development, including technical support, quality assurance, and documentation. He is also the creator of LinuxCommand.org, a Linux education and advocacy site featuring news, reviews, and extensive support for using the Linux command line.

SHELVE IN:

COMPUTERS/LINUX

$49.95 ($52.95 CDN)

www.nostarch.com

TH E F I N EST I N G E E K E NTE RTA I N M E NT™

FSC LOGO

“ I L I E F LAT .”

Th is book uses RepKover — a durab le b ind ing that won’t snap shut.

A C O M P L E T E I N T R O D U C T I O N

T H E L I N U XCO M M A N D L I N E

T H E L I N U XCO M M A N D L I N E

W I L L I A M E . S H O T T S , J R .

TH

E L

INU

X C

OM

MA

ND

LIN

ET

HE

LIN

UX

CO

MM

AN

D L

INE

SH

OT

TS

C O N T E N T S I N D E T A I L

ACKNOWLEDGMENTS xxiii

INTRODUCTION xxvWhy Use the Command Line?...................................................................................................xxviWhat This Book Is About..........................................................................................................xxviWho Should Read This Book....................................................................................................xxviiWhat’s in This Book................................................................................................................xxviiHow to Read This Book..........................................................................................................xxviiiPrerequisites..........................................................................................................................xxviii

PART 1LEARNING THE SHELL

1WHAT IS THE SHELL? 3Terminal Emulators......................................................................................................................3Your First Keystrokes....................................................................................................................4

Command History..........................................................................................................4Cursor Movement..........................................................................................................4

Try Some Simple Commands........................................................................................................5Ending a Terminal Session...........................................................................................................6

2NAVIGATION 7Understanding the Filesystem Tree.................................................................................................7The Current Working Directory.....................................................................................................8Listing the Contents of a Directory.................................................................................................8Changing the Current Working Directory.......................................................................................9

Absolute Pathnames.......................................................................................................9Relative Pathnames........................................................................................................9Some Helpful Shortcuts.................................................................................................10

The Linux Command Line © 2012 William E. Shotts, Jr.

3EXPLORING THE SYSTEM 13More Fun with ls.......................................................................................................................13

Options and Arguments................................................................................................14A Longer Look at Long Format.......................................................................................15

Determining a File’s Type with file...............................................................................................16Viewing File Contents with less...................................................................................................17A Guided Tour.........................................................................................................................19Symbolic Links..........................................................................................................................22

4MANIPULATING FILES AND DIRECTORIES 25Wildcards................................................................................................................................26mkdir—Create Directories..........................................................................................................28cp—Copy Files and Directories..................................................................................................28mv—Move and Rename Files.....................................................................................................30rm—Remove Files and Directories...............................................................................................31ln—Create Links........................................................................................................................32

Hard Links...................................................................................................................32Symbolic Links.............................................................................................................32

Let’s Build a Playground.............................................................................................................33Creating Directories.....................................................................................................33Copying Files..............................................................................................................33Moving and Renaming Files..........................................................................................34Creating Hard Links.....................................................................................................35Creating Symbolic Links................................................................................................36Removing Files and Directories......................................................................................37

Final Note................................................................................................................................38

5WORKING WITH COMMANDS 39What Exactly Are Commands?...................................................................................................40Identifying Commands...............................................................................................................40

type—Display a Command’s Type.................................................................................40which—Display an Executable’s Location.......................................................................41

Getting a Command’s Documentation.........................................................................................41help—Get Help for Shell Builtins....................................................................................41- -help—Display Usage Information.................................................................................42man—Display a Program’s Manual Page.......................................................................42apropos—Display Appropriate Commands.....................................................................43whatis—Display a Very Brief Description of a Command..................................................44info—Display a Program’s Info Entry..............................................................................44README and Other Program Documentation Files...........................................................45

Creating Your Own Commands with alias...................................................................................46Revisiting Old Friends................................................................................................................47

x Contents in Detail The Linux Command Line © 2012 William E. Shotts, Jr.

6REDIRECTION 49Standard Input, Output, and Error...............................................................................................50

Redirecting Standard Output.........................................................................................50Redirecting Standard Error............................................................................................51Redirecting Standard Output and Standard Error to One File............................................52Disposing of Unwanted Output......................................................................................52Redirecting Standard Input............................................................................................53

Pipelines..................................................................................................................................54Filters.........................................................................................................................55uniq—Report or Omit Repeated Lines.............................................................................55wc—Print Line, Word, and Byte Counts..........................................................................55grep—Print Lines Matching a Pattern..............................................................................56head/tail—Print First/Last Part of Files............................................................................56tee—Read from Stdin and Output to Stdout and Files.......................................................57

Final Note................................................................................................................................58

7SEEING THE WORLD AS THE SHELL SEES IT 59Expansion................................................................................................................................59

Pathname Expansion....................................................................................................60Tilde Expansion...........................................................................................................61Arithmetic Expansion....................................................................................................62Brace Expansion..........................................................................................................63Parameter Expansion....................................................................................................64Command Substitution..................................................................................................64

Quoting...................................................................................................................................65Double Quotes............................................................................................................65Single Quotes..............................................................................................................67Escaping Characters....................................................................................................67

Final Note................................................................................................................................68

8ADVANCED KEYBOARD TRICKS 69Command Line Editing...............................................................................................................70

Cursor Movement........................................................................................................70Modifying Text............................................................................................................70Cutting and Pasting (Killing and Yanking) Text................................................................70

Completion..............................................................................................................................72Using History............................................................................................................................73

Searching History........................................................................................................74History Expansion........................................................................................................75

Final Note................................................................................................................................76

Contents in Detail xiThe Linux Command Line © 2012 William E. Shotts, Jr.

9PERMISSIONS 77Owners, Group Members, and Everybody Else............................................................................78Reading, Writing, and Executing................................................................................................79

chmod—Change File Mode..........................................................................................81Setting File Mode with the GUI......................................................................................84umask—Set Default Permissions.....................................................................................84

Changing Identities...................................................................................................................87su—Run a Shell with Substitute User and Group IDs.........................................................87sudo—Execute a Command as Another User..................................................................88chown—Change File Owner and Group........................................................................90chgrp—Change Group Ownership................................................................................91

Exercising Your Privileges..........................................................................................................91Changing Your Password...........................................................................................................93

10PROCESSES 95How a Process Works...............................................................................................................96

Viewing Processes with ps............................................................................................96Viewing Processes Dynamically with top.........................................................................98

Controlling Processes...............................................................................................................100Interrupting a Process.................................................................................................101Putting a Process in the Background.............................................................................101Returning a Process to the Foreground..........................................................................102Stopping (Pausing) a Process.......................................................................................102

Signals..................................................................................................................................103Sending Signals to Processes with kill...........................................................................103Sending Signals to Multiple Processes with killall...........................................................106

More Process-Related Commands.............................................................................................106

PART 2CONFIGURATION AND THE ENVIRONMENT

11THE ENVIRONMENT 109What Is Stored in the Environment?...........................................................................................110

Examining the Environment.........................................................................................110Some Interesting Variables..........................................................................................111

How Is the Environment Established?.........................................................................................112Login and Non-login Shells.........................................................................................112What’s in a Startup File?............................................................................................113

xii Contents in Detail The Linux Command Line © 2012 William E. Shotts, Jr.

Modifying the Environment.......................................................................................................115Which Files Should We Modify?.................................................................................115Text Editors...............................................................................................................115Using a Text Editor.....................................................................................................116Activating Our Changes.............................................................................................118

Final Note..............................................................................................................................119

12A GENTLE INTRODUCTION TO VI 121Why We Should Learn vi.........................................................................................................122A Little Background.................................................................................................................122Starting and Stopping vi..........................................................................................................122Editing Modes........................................................................................................................123

Entering Insert Mode..................................................................................................124Saving Our Work......................................................................................................124

Moving the Cursor Around.......................................................................................................125Basic Editing..........................................................................................................................126

Appending Text.........................................................................................................127Opening a Line.........................................................................................................127Deleting Text.............................................................................................................128Cutting, Copying, and Pasting Text..............................................................................129Joining Lines..............................................................................................................131

Search and Replace................................................................................................................131Searching Within a Line.............................................................................................131Searching the Entire File.............................................................................................131Global Search and Replace........................................................................................132

Editing Multiple Files...............................................................................................................133Switching Between Files..............................................................................................134Opening Additional Files for Editing............................................................................134Copying Content from One File into Another.................................................................135Inserting an Entire File into Another..............................................................................136

Saving Our Work...................................................................................................................137

13CUSTOMIZING THE PROMPT 139Anatomy of a Prompt..............................................................................................................139Trying Some Alternative Prompt Designs....................................................................................141Adding Color.........................................................................................................................142Moving the Cursor..................................................................................................................144Saving the Prompt...................................................................................................................146Final Note..............................................................................................................................146

Contents in Detail xiiiThe Linux Command Line © 2012 William E. Shotts, Jr.

PART 3COMMON TASKS AND ESSENTIAL TOOLS

14PACKAGE MANAGEMENT 149Packaging Systems..................................................................................................................150How a Package System Works.................................................................................................150

Package Files............................................................................................................150Repositories...............................................................................................................151Dependencies............................................................................................................151High- and Low-Level Package Tools..............................................................................152

Common Package Management Tasks......................................................................................152Finding a Package in a Repository...............................................................................152Installing a Package from a Repository.........................................................................153Installing a Package from a Package File......................................................................153Removing a Package..................................................................................................154Updating Packages from a Repository..........................................................................154Upgrading a Package from a Package File...................................................................154Listing Installed Packages............................................................................................155Determining Whether a Package Is Installed.................................................................155Displaying Information About an Installed Package........................................................155Finding Which Package Installed a File........................................................................156

Final Note..............................................................................................................................156

15STORAGE MEDIA 159Mounting and Unmounting Storage Devices...............................................................................160

Viewing a List of Mounted Filesystems..........................................................................161Determining Device Names.........................................................................................164

Creating New Filesystems........................................................................................................167Manipulating Partitions with fdisk................................................................................167Creating a New Filesystem with mkfs...........................................................................169

Testing and Repairing Filesystems.............................................................................................170Formatting Floppy Disks...........................................................................................................171Moving Data Directly to and from Devices.................................................................................171Creating CD-ROM Images.......................................................................................................172

Creating an Image Copy of a CD-ROM........................................................................172Creating an Image from a Collection of Files................................................................172

Writing CD-ROM Images.........................................................................................................173Mounting an ISO Image Directly..................................................................................173Blanking a Rewritable CD-ROM...................................................................................173Writing an Image......................................................................................................173

Extra Credit............................................................................................................................174

xiv Contents in Detail The Linux Command Line © 2012 William E. Shotts, Jr.

16NETWORKING 175Examining and Monitoring a Network.......................................................................................176

ping—Send a Special Packet to a Network Host...........................................................176traceroute—Trace the Path of a Network Packet............................................................177netstat—Examine Network Settings and Statistics...........................................................178

Transporting Files over a Network.............................................................................................179ftp—Transfer Files with the File Transfer Protocol............................................................179lftp—A Better ftp........................................................................................................181wget—Non-interactive Network Downloader................................................................181

Secure Communication with Remote Hosts.................................................................................182ssh—Securely Log in to Remote Computers...................................................................182scp and sftp—Securely Transfer Files............................................................................185

17SEARCHING FOR FILES 187locate—Find Files the Easy Way...............................................................................................188find—Find Files the Hard Way.................................................................................................189

Tests.........................................................................................................................189Actions.....................................................................................................................194A Return to the Playground..........................................................................................198Options....................................................................................................................200

18ARCHIVING AND BACKUP 201Compressing Files...................................................................................................................202

gzip—Compress or Expand Files.................................................................................202bzip2—Higher Compression at the Cost of Speed.........................................................204

Archiving Files........................................................................................................................205tar—Tape Archiving Utility..........................................................................................205zip—Package and Compress Files...............................................................................209

Synchronizing Files and Directories...........................................................................................211rsync—Remote File and Directory Synchronization.........................................................212Using rsync over a Network........................................................................................213

19REGULAR EXPRESSIONS 215What Are Regular Expressions?................................................................................................216grep—Search Through Text......................................................................................................216Metacharacters and Literals......................................................................................................217The Any Character..................................................................................................................218Anchors.................................................................................................................................219

Contents in Detail xvThe Linux Command Line © 2012 William E. Shotts, Jr.

Bracket Expressions and Character Classes................................................................................220Negation..................................................................................................................220Traditional Character Ranges......................................................................................220POSIX Character Classes............................................................................................221

POSIX Basic vs. Extended Regular Expressions...........................................................................224Alternation.............................................................................................................................225Quantifiers.............................................................................................................................226

?—Match an Element Zero Times or One Time..............................................................226*—Match an Element Zero or More Times....................................................................227+—Match an Element One or More Times....................................................................227{ }—Match an Element a Specific Number of Times.......................................................228

Putting Regular Expressions to Work.........................................................................................229Validating a Phone List with grep.................................................................................229Finding Ugly Filenames with find.................................................................................230Searching for Files with locate.....................................................................................230Searching for Text with less and vim............................................................................231

Final Note..............................................................................................................................232

20TEXT PROCESSING 233Applications of Text.................................................................................................................234

Documents................................................................................................................234Web Pages...............................................................................................................234Email........................................................................................................................234Printer Output............................................................................................................234Program Source Code................................................................................................235

Revisiting Some Old Friends.....................................................................................................235cat—Concatenate Files and Print on Standard Output....................................................235sort—Sort Lines of Text Files........................................................................................236uniq—Report or Omit Repeated Lines...........................................................................242

Slicing and Dicing..................................................................................................................243cut—Remove Sections from Each Line of Files................................................................243paste—Merge Lines of Files.........................................................................................246join—Join Lines of Two Files on a Common Field...........................................................247

Comparing Text......................................................................................................................249comm—Compare Two Sorted Files Line by Line.............................................................249diff—Compare Files Line by Line..................................................................................250patch—Apply a diff to an Original..............................................................................253

Editing on the Fly....................................................................................................................254tr—Transliterate or Delete Characters...........................................................................254sed—Stream Editor for Filtering and Transforming Text...................................................256aspell—Interactive Spell Checker.................................................................................263

Final Note..............................................................................................................................266Extra Credit............................................................................................................................266

xvi Contents in Detail The Linux Command Line © 2012 William E. Shotts, Jr.

21FORMATTING OUTPUT 267Simple Formatting Tools...........................................................................................................268

nl—Number Lines......................................................................................................268fold—Wrap Each Line to a Specified Length.................................................................271fmt—A Simple Text Formatter......................................................................................271pr—Format Text for Printing........................................................................................274printf—Format and Print Data......................................................................................275

Document Formatting Systems...................................................................................................278The roff Family and TEX..............................................................................................279groff—A Document Formatting System..........................................................................279

Final Note..............................................................................................................................283

22PRINTING 285A Brief History of Printing.........................................................................................................286

Printing in the Dim Times.............................................................................................286Character-Based Printers.............................................................................................286Graphical Printers......................................................................................................287

Printing with Linux...................................................................................................................288Preparing Files for Printing.......................................................................................................288

pr—Convert Text Files for Printing................................................................................288Sending a Print Job to a Printer.................................................................................................290

lpr—Print Files (Berkeley Style).....................................................................................290lp—Print Files (System V Style).....................................................................................291Another Option: a2ps................................................................................................292

Monitoring and Controlling Print Jobs........................................................................................294lpstat—Display Print System Status...............................................................................294lpq—Display Printer Queue Status...............................................................................295lprm and cancel—Cancel Print Jobs.............................................................................296

23COMPILING PROGRAMS 297What Is Compiling?................................................................................................................298

Are All Programs Compiled?.......................................................................................299Compiling a C Program...........................................................................................................299

Obtaining the Source Code........................................................................................300Examining the Source Tree..........................................................................................301Building the Program..................................................................................................302Installing the Program.................................................................................................305

Final Note..............................................................................................................................306

Contents in Detail xviiThe Linux Command Line © 2012 William E. Shotts, Jr.

PART 4WRITING SHELL SCRIPTS

24WRITING YOUR FIRST SCRIPT 309What Are Shell Scripts?...........................................................................................................309How to Write a Shell Script......................................................................................................310

Script File Format.......................................................................................................310Executable Permissions...............................................................................................311Script File Location.....................................................................................................311Good Locations for Scripts..........................................................................................312

More Formatting Tricks............................................................................................................312Long Option Names...................................................................................................313Indentation and Line Continuation................................................................................313

Final Note..............................................................................................................................314

25STARTING A PROJECT 315First Stage: Minimal Document.................................................................................................315Second Stage: Adding a Little Data...........................................................................................317Variables and Constants..........................................................................................................318

Creating Variables and Constants................................................................................318Assigning Values to Variables and Constants................................................................320

Here Documents.....................................................................................................................321Final Note..............................................................................................................................323

26TOP-DOWN DESIGN 325Shell Functions........................................................................................................................326Local Variables.......................................................................................................................328Keep Scripts Running...............................................................................................................330Final Note..............................................................................................................................332

27FLOW CONTROL: BRANCHING WITH IF 333Using if..................................................................................................................................334Exit Status..............................................................................................................................334Using test...............................................................................................................................336

File Expressions.........................................................................................................336String Expressions......................................................................................................338Integer Expressions....................................................................................................340

xviii Contents in Detail The Linux Command Line © 2012 William E. Shotts, Jr.

A More Modern Version of test.................................................................................................341(( ))—Designed for Integers......................................................................................................342Combining Expressions............................................................................................................343Control Operators: Another Way to Branch...............................................................................345Final Note..............................................................................................................................346

28READING KEYBOARD INPUT 347read—Read Values from Standard Input....................................................................................348

Options....................................................................................................................351Separating Input Fields with IFS...................................................................................351

Validating Input......................................................................................................................353Menus...................................................................................................................................355Final Note..............................................................................................................................356Extra Credit............................................................................................................................356

29FLOW CONTROL: LOOPING WITH WHILE AND UNTIL 357Looping.................................................................................................................................358while.....................................................................................................................................358Breaking out of a Loop............................................................................................................360until.......................................................................................................................................361Reading Files with Loops..........................................................................................................362Final Note..............................................................................................................................362

30TROUBLESHOOTING 363Syntactic Errors.......................................................................................................................363

Missing Quotes.........................................................................................................364Missing or Unexpected Tokens....................................................................................365Unanticipated Expansions...........................................................................................365

Logical Errors.........................................................................................................................366Defensive Programming..............................................................................................367Verifying Input...........................................................................................................368

Testing...................................................................................................................................369Stubs........................................................................................................................369Test Cases.................................................................................................................369

Debugging.............................................................................................................................370Finding the Problem Area...........................................................................................370Tracing.....................................................................................................................371Examining Values During Execution.............................................................................373

Final Note..............................................................................................................................373

Contents in Detail xixThe Linux Command Line © 2012 William E. Shotts, Jr.

31FLOW CONTROL: BRANCHING WITH CASE 375case......................................................................................................................................376

Patterns.....................................................................................................................377Combining Multiple Patterns........................................................................................378

Final Note..............................................................................................................................379

32POSITIONAL PARAMETERS 381Accessing the Command Line...................................................................................................381

Determining the Number of Arguments.........................................................................382shift—Getting Access to Many Arguments.....................................................................383Simple Applications...................................................................................................384Using Positional Parameters with Shell Functions............................................................385

Handling Positional Parameters En Masse..................................................................................385A More Complete Application..................................................................................................387Final Note..............................................................................................................................390

33FLOW CONTROL: LOOPING WITH FOR 393for: Traditional Shell Form........................................................................................................393for: C Language Form.............................................................................................................396Final Note..............................................................................................................................397

34STRINGS AND NUMBERS 399Parameter Expansion...............................................................................................................399

Basic Parameters.......................................................................................................400Expansions to Manage Empty Variables.......................................................................400Expansions That Return Variable Names.......................................................................401String Operations......................................................................................................402

Arithmetic Evaluation and Expansion.........................................................................................404Number Bases...........................................................................................................405Unary Operators.......................................................................................................405Simple Arithmetic.......................................................................................................405Assignment...............................................................................................................406Bit Operations...........................................................................................................408Logic........................................................................................................................409

bc—An Arbitrary-Precision Calculator Language........................................................................411Using bc...................................................................................................................412An Example Script.....................................................................................................413

Final Note..............................................................................................................................414Extra Credit............................................................................................................................414

xx Contents in Detail The Linux Command Line © 2012 William E. Shotts, Jr.

35ARRAYS 415What Are Arrays?...................................................................................................................415Creating an Array...................................................................................................................416Assigning Values to an Array...................................................................................................416Accessing Array Elements........................................................................................................417Array Operations....................................................................................................................418

Outputting the Entire Contents of an Array....................................................................419Determining the Number of Array Elements...................................................................419Finding the Subscripts Used by an Array......................................................................420Adding Elements to the End of an Array.......................................................................420Sorting an Array........................................................................................................420Deleting an Array......................................................................................................421

Final Note..............................................................................................................................422

36EXOTICA 423Group Commands and Subshells..............................................................................................423

Performing Redirections..............................................................................................424Process Substitution....................................................................................................424

Traps.....................................................................................................................................426Asynchronous Execution..........................................................................................................429

wait.........................................................................................................................429Named Pipes.........................................................................................................................430

Setting Up a Named Pipe...........................................................................................431Using Named Pipes...................................................................................................431

Final Note..............................................................................................................................432

INDEX 433

Contents in Detail xxiThe Linux Command Line © 2012 William E. Shotts, Jr.