Unix Programming Environment Part 2 – An Introduction to Unix Systems Dept. of CSE BUAA.

Unix Programming Unix Programming EnvironmentEnvironment

Part 2 – An Introduction to Unix Systems

Dept. of CSEDept. of CSEBUAABUAA

Unix Programming EnvironmentUnix Programming Environment Dept. of CSE, BUAADept. of CSE, BUAA

AgendaAgenda 1. Overview1. Overview

2. A Brief History of Unix Systems2. A Brief History of Unix Systems

3. Organizations and Standards about Unix3. Organizations and Standards about Unix

4. Unix Nowadays 4. Unix Nowadays

Internal structure of modern Unix, system interfaces, and toolsInternal structure of modern Unix, system interfaces, and tools

5. Summary - Unix’s legend5. Summary - Unix’s legend

6. The mandate for change6. The mandate for change

7. Looking back, looking forward7. Looking back, looking forward

8. Supplementary Readings8. Supplementary Readings


1. Overview (1)1. Overview (1) Since its inception in 1969, the Unix system has grown into a Since its inception in 1969, the Unix system has grown into a

versatile operating platform: versatile operating platform:

small embedded processorssmall embedded processors

workstations and desktop systemsworkstations and desktop systems

high-performance multi-processor systemshigh-performance multi-processor systems

““The UNIX systemThe UNIX system”” consists of a collection of consists of a collection of user programs, user programs, libraries, and utilitieslibraries, and utilities, running on , running on the UNIX operating system kernelthe UNIX operating system kernel, , which provides a run-time environment and system services for which provides a run-time environment and system services for these applications.these applications.

Baseline releases: Baseline releases:

System V Release 4( SVR4)System V Release 4( SVR4)

UC’s Berkeley Software Distribution (4.xBSD )UC’s Berkeley Software Distribution (4.xBSD )

LinuxLinux

CMU’s MachCMU’s Mach


1. Overview (2) - How UNIX is organized1. Overview (2) - How UNIX is organized

The UNIX system is functionally The UNIX system is functionally organized at three levels:organized at three levels:

The kernel, The kernel, which schedules which schedules tasks and manages storagetasks and manages storage; ;

The shell, The shell, which connects and which connects and interprets users' commands, calls interprets users' commands, calls programs from memory, and programs from memory, and executes them;executes them;

The tools and applications The tools and applications that that offer additional functionality to the offer additional functionality to the operating system operating system

Tools can be added or removed Tools can be added or removed from a UNIX system, depending from a UNIX system, depending upon the applications required. upon the applications required.


2. A Brief History of Unix (1)2. A Brief History of Unix (1) 2.1 The Beginning2.1 The Beginning

In the late 1960s, BTL & GE & MIT: In the late 1960s, BTL & GE & MIT:

Multics: Multics: MultiMultiplexed plexed IInformation and nformation and CComputing omputing SSystemystem

One of the principal developers, Ken Thompsons, Game program One of the principal developers, Ken Thompsons, Game program called “Space Travel”: called “Space Travel”:

PDP-7: lacking a program development environmentPDP-7: lacking a program development environment

Honeywell 635 running the GECOS: cross-assemblerHoneywell 635 running the GECOS: cross-assembler

Ken Thompsons & Dennis Ritchie: develop an operating Ken Thompsons & Dennis Ritchie: develop an operating environment for PDP-7environment for PDP-7

A file system: the ancestor of s5fsA file system: the ancestor of s5fs

A process subsystem: like “fork”A process subsystem: like “fork”

A simple command interpreter called the “A simple command interpreter called the “shellshell””

Brian Kernighan called this system as “Unics”( Uniplexed Brian Kernighan called this system as “Unics”( Uniplexed Information and Computing system) => “UNIX”Information and Computing system) => “UNIX”


2. A Brief History of Unix (2)2. A Brief History of Unix (2)


2. A Brief History of Unix (2)2. A Brief History of Unix (2) Develop a text-processing system for the patent department at BTL: Develop a text-processing system for the patent department at BTL:

Port UNIX to the PDP-11Port UNIX to the PDP-11

Several text-processing utilities including “ed”, “runoff”, etcSeveral text-processing utilities including “ed”, “runoff”, etc

A new language called “B”:A new language called “B”:

BCPL, CPL, Algol60BCPL, CPL, Algol60

B: an interpretive languageB: an interpretive language

To improve the performance, Ritchie evolved it into “C”: compilable and supporting To improve the performance, Ritchie evolved it into “C”: compilable and supporting data types and data structuresdata types and data structures

1973 – 1974: having a tremendous impact on Unix’s future success1973 – 1974: having a tremendous impact on Unix’s future success

Unix Programmer ManualUnix Programmer Manual

Unix was rewritten in C( resulting in v4 )Unix was rewritten in C( resulting in v4 )

The first Unix paper on ACM SOSP – “The Unix Time Sharing The first Unix paper on ACM SOSP – “The Unix Time Sharing System”( Supplementary reading )System”( Supplementary reading )

Distributing Unix to outside including its source codeDistributing Unix to outside including its source code



2.2 Proliferation2.2 Proliferation

As a result of antitrust litigation by the Dept. of Justice, AT&T As a result of antitrust litigation by the Dept. of Justice, AT&T distributed Unix for educational and research purpose royalty-distributed Unix for educational and research purpose royalty-free: one of such licensees was U.C. at Berkeley at 1973. free: one of such licensees was U.C. at Berkeley at 1973.

Mel Ferentz: <Unix NewsLetter> => USENIX ConferenceMel Ferentz: <Unix NewsLetter> => USENIX Conference

John Lions from Australia: lecture notes => <A commentary on John Lions from Australia: lecture notes => <A commentary on the UNIX operating System>the UNIX operating System>

Many people ported UNIX to various architectures: Interdata 8/32, Many people ported UNIX to various architectures: Interdata 8/32, Interdata 7/31, VM/370, VAX-11( UNIX/32V ), Intel 8086( by SCO & Interdata 7/31, VM/370, VAX-11( UNIX/32V ), Intel 8086( by SCO & MS ), etcMS ), etc

The spirit of cooperation between its keepers and users was a The spirit of cooperation between its keepers and users was a key factor in the rapid growth and rising popularity of UNIX.key factor in the rapid growth and rising popularity of UNIX.

Version 7 Unix, released in 1979, was the first truly portable Version 7 Unix, released in 1979, was the first truly portable UNIX system. UNIX system.



2.3 Baseline Release 1: BSD2.3 Baseline Release 1: BSD

Utilities: ex->vi, a Pascal compiler, C Shell( job control, command Utilities: ex->vi, a Pascal compiler, C Shell( job control, command history )history )

1978: VAX-11/780( 32-bit architecture, 4G address space, but only 1978: VAX-11/780( 32-bit architecture, 4G address space, but only 2M physical memory) => a paging-based virtual memory system 2M physical memory) => a paging-based virtual memory system for UNIXfor UNIX

The Defense Advanced Research Projects Agency( DARPA ) The Defense Advanced Research Projects Agency( DARPA ) funded the development of Unix systems at Berkeley: funded the development of Unix systems at Berkeley:

Integrating TCP/IP network protocol suite => InternetIntegrating TCP/IP network protocol suite => Internet

4.4BSD is the last release from CSRG at Berkeley. 4.4BSD is the last release from CSRG at Berkeley.

FreeBSD and OpenBSD, basing on 4.4BSD, still evolve now. FreeBSD and OpenBSD, basing on 4.4BSD, still evolve now.

Contributions from BSD: VM, TCP/IP, Fast file system( FFS), a Contributions from BSD: VM, TCP/IP, Fast file system( FFS), a reliable signals implementation, and the sockets facility, a log-reliable signals implementation, and the sockets facility, a log-structured file system, etcstructured file system, etc



2.4 Baseline Release 2: System V2.4 Baseline Release 2: System V

After “baby Bells” were born, AT&T was allowed to enter the After “baby Bells” were born, AT&T was allowed to enter the computer business in 1982. computer business in 1982.

AT&T marketed Unix aggressively: System III at 1982, System V AT&T marketed Unix aggressively: System III at 1982, System V in 1983, SVR2 in 1985, SVR3 in 1987:in 1983, SVR2 in 1985, SVR3 in 1987:

A VM implementation quite different from that of BSD: A VM implementation quite different from that of BSD: “regions”(!!!)“regions”(!!!)

System V IPC: shared memory, semaphores and message System V IPC: shared memory, semaphores and message queuesqueues

The STREAMS framework for device drivers and network The STREAMS framework for device drivers and network protocols: XTIprotocols: XTI

Shared libraries, remote file sharing, etcShared libraries, remote file sharing, etc


2. A Brief History of Unix (6)2. A Brief History of Unix (6) SVR4, developed jointly by AT&T and Sun, was first released in SVR4, developed jointly by AT&T and Sun, was first released in

1989:1989:

Integrating features from SVR3, 4BSD, SunOS and XENIXIntegrating features from SVR3, 4BSD, SunOS and XENIX

Virtual File System( VFS ) and Virtual Memory( VM ) from SunVirtual File System( VFS ) and Virtual Memory( VM ) from Sun

Real-time scheduling classesReal-time scheduling classes

Supporting Symmetric Multiple Processors( SMP )Supporting Symmetric Multiple Processors( SMP )

Kernel-level threadsKernel-level threads

Fine-grained locksFine-grained locks



2.4 Baseline Release 3: Mach2.4 Baseline Release 3: Mach

The Unix kernel is small and simple, yet offered many useful The Unix kernel is small and simple, yet offered many useful facilities. As more and more features were incorporated into the facilities. As more and more features were incorporated into the kernel, the internal elegancy disappeared. kernel, the internal elegancy disappeared.

The microkernel approach: export a few simple abstractions, do The microkernel approach: export a few simple abstractions, do provide most of the functionality thorough a collection of user-provide most of the functionality thorough a collection of user-level tasks. level tasks.

But the performance becomes the critical hurdle for the But the performance becomes the critical hurdle for the “microkernel” architecture. “microkernel” architecture.


3. Organizations and Standards3. Organizations and Standards Organization: Organization:

UI & OSFUI & OSF

Unix International( UI ): AT&T and Sun, System V-based Unix International( UI ): AT&T and Sun, System V-based systemssystems

Open Software Foundation( OSF ): Motif, OSF/1, DCEOpen Software Foundation( OSF ): Motif, OSF/1, DCE

USENIX: USENIX:

But the growth, and even survival, of UNIX has been jeopardized But the growth, and even survival, of UNIX has been jeopardized by Microsoft Windows. by Microsoft Windows.

Standards:Standards:

System V Interface Description( SVID ) from USLSystem V Interface Description( SVID ) from USL

POSIX from IEEEPOSIX from IEEE

XPG from X/OpenXPG from X/Open


4. Unix Nowadays (1)4. Unix Nowadays (1)

Internal Structure of Traditional UnixInternal Structure of Traditional Unix



Internal Structure of modern UnixInternal Structure of modern Unix


4. Unix Nowadays (4)4. Unix Nowadays (4) SummarySummary

Architecture: Functions, Architecture: Functions, Internal OrganizationInternal Organization

Dynamic loadingDynamic loading


4. Unix Nowadays (5) - Standards (1)4. Unix Nowadays (5) - Standards (1)

Unix



X/Open Common Applications Environment (CAE) X/Open Common Applications Environment (CAE) Portability Guide Issue 3 (XPG3) and Issue 4 (XPG4)Portability Guide Issue 3 (XPG3) and Issue 4 (XPG4)

SUS( Single UNIX Specification ), SUSv2SUS( Single UNIX Specification ), SUSv2

XNS4: Networking Services Issue4XNS4: Networking Services Issue4

http://www.unix-systems.orghttp://www.unix-systems.org

Notes:Notes:

1. The developers of SVID3( UNIX Systems Laboratories) are no 1. The developers of SVID3( UNIX Systems Laboratories) are no longer in business, and this specification defers to POSIX and longer in business, and this specification defers to POSIX and X/Open CAE. X/Open CAE.

2. POSIX Standards: we can get them from the library. 2. POSIX Standards: we can get them from the library.



XPG.3

POSIX.2

POSIX.1b

POSIX.1c

BSD Interfaces

SUS( XPG4v2)

ISO-C Amendement 1

XNS4

SUSV2XNS5

LP64-clean derivative of XNS4

POSIX Standard

XPG.4

POSIX.1

SVR4

Solaris8

XPG.3

SUSV2


4. Unix Nowadays (8)4. Unix Nowadays (8) Software Development Toolset for UnixSoftware Development Toolset for Unix

GNU Development Toolset – gcc, ld, gdb, building systemGNU Development Toolset – gcc, ld, gdb, building system


5. Unix’s Legend (1)5. Unix’s Legend (1)


5. Unix’s Legend (2)5. Unix’s Legend (2)

Linus Torvalds Bill Joy Ken Thompson Dennis Ritchie Brian Kernighan

Unix Creators & Gurus


What’s GNU?What’s GNU?

Richard Stallman – the founder and author of many GNU programsFree Software FoundationFree Software Foundation

GPL - GPL - Copyright is called…Copyleft.

Linux => GNU/Linux


6. The Mandate for Change6. The Mandate for Change 1. Functionalities: 1. Functionalities:

Pipe -> System V IPC -> POSIX IPCPipe -> System V IPC -> POSIX IPC

MultithreadMultithread

2. Networking2. Networking

TCP/IP Protocol StackTCP/IP Protocol Stack

Distributed File System: NFS from Sun, AFS/Coda from CMUDistributed File System: NFS from Sun, AFS/Coda from CMU

RPC-based Services: NIS from Sun, DCE from OSFRPC-based Services: NIS from Sun, DCE from OSF

3. Performance3. Performance

FFS, Log FSFFS, Log FS

SMP ArchitectureSMP Architecture

4. Computation Paradigm Shifts4. Computation Paradigm Shifts

A Host + terminals – networked workstations – C/S model – A Host + terminals – networked workstations – C/S model – Distributed object/component-based computationDistributed object/component-based computation


7. Looking back, Looking forward( 1 )7. Looking back, Looking forward( 1 ) 1. “Small is beautiful”: 1. “Small is beautiful”:

2. Flexibility: the core is a framework2. Flexibility: the core is a framework

Simplicity, clarity and flexibility Simplicity, clarity and flexibility

Methodology: combination, interface-oriented Methodology: combination, interface-oriented

3. what’s wrong with Unix? 3. what’s wrong with Unix?

Although initially simple, it did not remain the way: Standard I/OAlthough initially simple, it did not remain the way: Standard I/O

No uniform user interfacesNo uniform user interfaces

Ritche said: “Unix is simple and coherent, but it takes a genius to Ritche said: “Unix is simple and coherent, but it takes a genius to understand and appreciate its simplicity.”understand and appreciate its simplicity.”

The building-block approach requires creativity and imagination to The building-block approach requires creativity and imagination to use effectively. use effectively.

cat menu | grep shrimp | test -lt $10cat menu | grep shrimp | test -lt $10

Because of too many value-added features, Unix became chaotic. Because of too many value-added features, Unix became chaotic. Standards like POSIX are partially successful now. Standards like POSIX are partially successful now.


7. Looking back, Looking forward( 2 )7. Looking back, Looking forward( 2 ) Philosophy Matters and Traditions in UnixPhilosophy Matters and Traditions in Unix

Lessons of Unix can be applied in our programming practiceLessons of Unix can be applied in our programming practice

‘‘KISS Principle’KISS Principle’: : Keep it simple, stupid! / Simplicity, clarity and generalityKeep it simple, stupid! / Simplicity, clarity and generality

Write small pieces connected by clean interfaces. Write small pieces connected by clean interfaces.

Design programs to communicate easily with other programs. Design programs to communicate easily with other programs.

Robustness is the child of transparency and simplicity. Robustness is the child of transparency and simplicity.

Design for simplicity; add complexity only where you must. Design for simplicity; add complexity only where you must.

Design for transparency; spend effort early to save effort later. Design for transparency; spend effort early to save effort later.

In interface design, obey the Rule of Least Surprise. In interface design, obey the Rule of Least Surprise.

Programmer time is expensive; conserve it in preference to machine Programmer time is expensive; conserve it in preference to machine time. time.

Avoid hand-hacking; write programs to write programs when you Avoid hand-hacking; write programs to write programs when you can. can.

Use smart data so program logic can be stupid and robust. Use smart data so program logic can be stupid and robust.

Prototype, then polish. Get it working before you optimize it.Prototype, then polish. Get it working before you optimize it.


7. Looking back, Looking forward( 3 )7. Looking back, Looking forward( 3 ) Everything that can be a device-independent filter Everything that can be a device-independent filter shouldshould be. be.

Data streams should, if at all possible, be textual (so they can Data streams should, if at all possible, be textual (so they can be viewed and filtered with standard tools). be viewed and filtered with standard tools).

Database layouts should if at all possible be textual (human-Database layouts should if at all possible be textual (human-editable). editable).

Application protocols should if at all possible be textual Application protocols should if at all possible be textual (human-readable). (human-readable).

Complex front ends (user interfaces) should be cleanly Complex front ends (user interfaces) should be cleanly separated from complex back ends. separated from complex back ends.

Whenever possible, prototype in an interpretive language Whenever possible, prototype in an interpretive language before coding C. before coding C.

Mixing languages is better than writing everything in one, if Mixing languages is better than writing everything in one, if using only that one will over-complicate the program. using only that one will over-complicate the program.


8. Supplementary Readings8. Supplementary Readings

The UNIX Time-Sharing System The UNIX Time-Sharing System Dennis M. Ritchie and Ken Thompson Dennis M. Ritchie and Ken Thompson

Early Unix history and evolutionEarly Unix history and evolutionDennis M. Ritchie Dennis M. Ritchie

Themes in Operating Systems Themes in Operating Systems

Dennis M. Ritchie Dennis M. Ritchie

25th Anniversary of Unix 25th Anniversary of Unix

An introduction to SolarisAn introduction to Solaris

Unix Programming Environment Part 2 – An Introduction to Unix Systems Dept. of CSE BUAA.

Documents

Transcript of Unix Programming Environment Part 2 – An Introduction to Unix Systems Dept. of CSE BUAA.