Operating Systems and System Programming

MIDORI SUGAYA SHIBAURA INSTITUTE OF TECHNOLOGY

INFORMATION SCIENCE AND ENGINEERING.

1

2013/09/30

Course Structure Place

5142 (5 Build. 1F, PC Training Room) Course Schedule

Totally 14 sessions First two sessions

Learn basic idea and concept of operating systems Learn basic command for Linux Install Linux for your computer (Using virtual machine)

The remaining sessions Learn detailed mechanism of operating systems and

system call programming Course Materials

http://www.sic.shibaura-it.ac.jp/~doly/os/ http://www.dlab.ise.shibaura-it.ac.jp/lectures.html

2

2013/09/30

Course Schedule

Week Topic

1-2 (OS) Introduction and Overview, Operating Systems Structure (Includes what is system call)

(SysPro) Administrative, Basic Commands3-5 (OS) Process Management Scheduling, Process and Threads,

Process Synchronization Deadlocks

(SysPro) Process, Thread, Synchronization, C and deep C6-8 (OS) Memory management : Main memory and virtual memory,

File-System Interface,

(SysPro) Memory, File operations,

9-11 (OS) Process Communication

(SysPro) (1) Signal (2) Socket

12-13 (OS) Security, Protection(SysPro) Security

3

2013/09/29

For Computer Science Students In first two session

I think it is necessary to explain the summary of OS before the system call programing.

For computer science students Project1: Recompile kernel and install it.

Please recompile your kernel from kernel-source. Download the original kernel

Do the lspci command to check your device Project2: Research the sequence of system call in Linux

How to trap kernel mode How to use the service of kernel How to return user mode

Project3: Adding a System Call to the Linux Kernel, p.74 Getting Started Building a New Kernel Extending Kernel Source Using the System Call From a User Program

4

2013/09/29

Course Materials Required Text

Operating System Concepts, 8 and 9th edition Silberschatz, Galvin, & Gagne

Multiple Cheap Options exist! Amazon/eBay

Operating Systems; Principles & Practice Anderson & Dahlin

Main differences More focus on typical modern kernels and hardware Less focus on historical systems Stronger coverage of file systems Weaker coverage of security

6

2013/09/30

Materials

7

Understanding UNIX/LINUX Programming: A Guide to Theory and Practice, Bruce Molay, Prentice Hall; 1st version (2002/11/25), 11,200

2013/09/29

Basic Command Login

Linux 1.2.13 (maya) (ttyp1) maya login: doly Password: *******

Logout $exit

Read manual $ man

8

2013/09/29

Todays Contents

Operating systems introduction

2013/09/29

9

Home ICT

Computing Environment

Health Care

Energy Saving

Public facilities

Electric generating facilities

Data Center Super Computer

Cloud Internet

Smart Phone

Display Touch Panel Network Communications

Smart Grid

Very Large Scale Computer

Embedded Systems

Programming Education

Sensor Network

Robot

Home ICT

Expected Computing Environment

Health Care

Energy Saving

Public facilities

Electric generating facilities

Data Center Super Computer

Cloud Internet

Smart Phone

Display Touch Panel Network Communications

Smart Grid

Very Large Scale Computer

Embedded Systems

Programming Education

Sensor Network

Robot

Behavior of userControl Robots

Smart Grid

Advanced Control through the use of data

Home ICT

Expected Computing Environment

Health Care

Energy Saving

Public facilities

Electric generating facilities

Data Center Super Computer

Cloud Internet

Smart Phone

Display Touch Panel Network Communications

Smart Grid

Very Large Scale Computer

Embedded Systems

Programming Education

Sensor Network

Robot

Behavior of userControl Robots

Smart Grid

Advanced Control through the use of data

Year 2000-Ubiquitous Year 2010-Smart GridCPS (Cyber Physical Computing) basic knowledge of system architecture (operating systems) is always required

What operating systems Do For Users, For Applications, etc.. Abstract view of the components of a computer systems

13

2013/09/29

What Operating Systems Do Computer = HW + OS + Apps + Users

Provide abstraction of hardware to use its services OS serves as interface between HW and (Apps & Users) OS provides service for Apps & Users Possible to write program independent from machine

architecture

Manages resources Like a middle manager OS provides the way to share the limited resources in computer

systems Ordering Scheduling Avoid resource abuse or monopolize Protection

14

2013/09/29

Computer-System Organization What are all the parts, and how do they fit together?

15

2013/09/29

Computer-System Operation Bootstrap program

Initialization of the system (from CPU registers to device controllers).

Load operating system and start execution $ dmesg

Interrupt From hardware

Trigger an interrupt at any time by sending a signal to CPU, by way of system bus $ cat /proc/interrupts

From software May trigger an interrupt by executing a special operation called

a system call

16

2013/09/29

Storage Structure Main memory (RAM)

Programs must be loaded into RAM to run Instructions and data fetched from RAM into registers RAM is volatile Medium size and speed

$ free Other electronic (volatile) memory is faster, smaller, and more

expensive per bit: Registers CPU cache

$ cat /proc/cpuinfo (cache size) Non-volatile memory (permanent storage) is slower, larger, and

less expensive per bit: Solid-state disk Magnetic disks Optical disks

$ df -h

18

2013/09/29

Storage-device hierarchy hierarchy

19

2013/09/29

I/O Structure Typical operation involves I/O requests, direct memory access (DMA),

and interrupt handling.

How modern computer system works $ cat /proc/dma

20

2013/09/29

Computer-System Architecture Different Operating Systems for Different Kinds of Computer

Environments

Single-Processor Systems One main CPU which manages the computer and runs user

apps. Other specialized processors ( disk controllers, GPUs, etc. ) do

not run user apps. Multiprocessor Systems

Increased throughput - Faster execution, but not 100% linear speedup.

Economy of scale - They can share peripherals, disks, memory, shared among processors.

Increased reliability If functions can be distributed properly among several processers, a failure of a CPU will not halt the system, only slows system, doesn't crash it.

21

2013/09/29

Multi-Processor Systems Two types of multiprocessor systems in use today.

Asymmetric multiprocessing architecture Each processor is assigned a specific task

Symmetric multiprocessing (SMP) architecture Each processor performs all tasks within the operating systems

A dual core design with two cores places on the same machine.

22

2013/09/29

Clustered Systems Another type of multiple CPU systems - Independent systems

Generally accepted definition: shared common storage and connected by a high-speed LAN,

working together. Clustering is usually used to provide high-availability service;

That is, service will continue even if one or more systems in the cluster fail.

General structure of a clustered system

23

2013/09/29

Operating System Structure Multiprogram

A single user can not keep either the CPU or the I/O devices bus at all times.

The idea : OS keeps several jobs in memory simultaneously

The job may have to wait for some task, such as an I/O operation, to complete

Time-sharing (or multi-tasking) Logical extension of multiprogramming OS requires:

Memory management Process management Job scheduling Resource allocation strategies Swap space / virtual memory in physical memory Interrupt handling File system management Protection and security Inter-process communications

24

2013/09/29

Operating-System Operations Modern operating systems are interrupt driven.

Events are always signaled by the occurrence of an interrupt or trap (or an exception).

Trap: software generated interrupt caused by an error (invalid memory access, etc..) or by a specific request from a user program.

For each type of interrupt, interrupt handler determine what type of action should be taken.

OS and users share the hardware and software resources. Need make sure that an error in a user program could cause

problems. E.g. if a process gets stuck in an infinite loop, this loop could

prevent the correct operation of many other programs, or even operating system itself.

25

2013/09/29

Dual-Mode and Multimode Operation User mode

when executing harmless code in user applications Kernel mode( a.k.a. system mode, supervisor mode, privileged

mode )

when executing potentially dangerous code in the system kernel.

Certain machine instructions ( privileged instructions ) can only be executed in kernel mode.

Kernel mode can only be entered by making system calls. User code cannot flip the mode switch.

Modern computers support dual-mode operation in hardware, and therefore most modern OSes support dual-mode operation.

26

2013/09/29

Transition from user to kernel mode User Kernel

User mode Kernel mode (supervisor mode, system mode, privileged mode)

With the mode, we are able to distinguish between a task that is executed on behalf of the OS or the user.

Hardware allows privileged instructions to be executed only in kernel mode. Check $ time ./a.out

System calls provide the means for a user program to ask the operating system on the user programs behalf. 2

7

2013/09/29

Process Management A program in execution is a process. A program by itself is not a process.

passive entity, file stored on disk A process

Active entity, a single-threaded process has one program counter specifying the next instruction to execute.

A process needs certain resources to accomplish its task. CPU time, memory, files, and I/O devices These resources are either given to the process when it is created or allocated to it

while it is running. Check $ top

All these processes can potentially execute concurrently by multiplexing the CPU among then on a single CPU, for example.

An OS is responsible for the following tasks with regards to process management:

Creating and deleting both user and system processes Ensuring that each process receives its necessary resources, without interfering

with other processes. Suspending and resuming processes Process synchronization and communication Deadlock handling In system call programming section, we will develop a process. 2

8

2013/09/29

Memory Management Main Memory

A repository of quickly accessible data shared by the CPU and I/O devices.

The main memory is generally the only large storage device that CPU is able to address and access directly access.

To improve both the utilization of the CPU and the speed of the computers response to its users, general-purpose computers must keep several programs in memory, creating a need for memory management.

An OS is responsible for the following tasks with regards to memory management:

Keeping track of which blocks of memory are currently in use, and by which processes.

Determining which blocks of code and data to move into and out of memory, and when.

Allocating and deallocating memory as needed. ( E.g. new, malloc )

29

2013/09/29

Storage Management To make the computer system convenient for users, the OS provides

a uniform, logical view of information storage.

The OS abstracts from the physical properties of its storage devices to define a logical storage unit, the file. The OS maps files onto physical media and access these files via the storage devices.

OS maps files onto physical media and access these files via the storage devices

An OS is responsible for the following tasks with regards tofile-system management:

Creating and deleting files and directories Supporting primitives for manipulating files and directories.

( open, flush, etc. ) Mapping files onto secondary storage. Backing up files onto stable permanent storage media.

30

2013/09/29

File-System Management Computers can store information on several different types of

physical media

Magnetic disk Optical disk Magnetic tape Solid state disk etc.. Each medium is controlled by a device, that also has its own

unique characteristic. Access speed, capacity, data-transfer rate, access method

31

2013/09/29

Directory Tree Unix organize file abstraction as tree

provides directories and files. In this organization, user can find unique file, use the command that move to the different tree node, and read/write the contents of file.

Root level

/etc, /home, /binetc User data /home/user

Operation command ls list cd change directory pwd print working directory

32

/

etc home bin dev var usr

tmp spool

mail fax

2013/09/30

Caching Caching is an important principle of computer systems.

Because cache has limited size, cache management is an important design problem.

Careful selection of the cache size and of a replacement policy can result in greatly increased performance.

33

Slower Storagecache

2013/09/29 Performance of various level of storage

Caching In a hierarchical storage structure, the same data may appear in

different levels of the storage system.

Data read in from disk follows a migration path from the hard drive to main memory, then to the CPU cache, and finally to the registers before it can be used, while data being written follows the reverse path.

2013/09/29

34

I/O Systems The I/O subsystem consists of several components:

A memory-management component that includes buffering, caching, and spooling.

A general device-driver interface. Drivers for specific hardware devices.

$ cd /dev/ $ ls l /dev/

2013/09/29

35

Protection and Security Protection

involves ensuring that no process access or interfere with resources to which they are not entitled, either by design or by accident. ( E.g. "protection faults" when pointer variables are misused. )

Security involves protecting the system from deliberate attacks, either

from legitimate users of the system attempting to gain unauthorized access and privileges, or external attackers attempting to access or damage the system.

2013/09/29

36

Computing Environment Mobile Computing

Computing on small handheld devices such as smart phones or tablets. ( As opposed to laptops, which still fall under traditional computing. )

May take advantage of additional built-in sensors, such as GPS, tilt, compass, and inertial movement.

Typically connect to the Internet using wireless networking ( IEEE 802.11 ) or cellular telephone technology.

Limited in storage capacity, memory capacity, and computing power relative to a PC.

Generally uses slower processors, that consume less battery power and produce less heat.

The two dominant OSes today are Google Android and Apple iOS.

2013/09/29

37

Distributed Systems A collection of physically separate, consists of multiple, possibly

heterogeneous, computers connected together via a network and cooperating in some way, form, or fashion.

Networks may range from small tight LANs to broad reaching WANs. WAN = Wide Area Network, such as an international corporation MAN =Metropolitan Area Network, covering a region the size of a

city for example. LAN =Local Area Network, typical of a home, business, single-site

corporation, or university campus. PAN = Personal Area Network, such as the bluetooth connection

between your PC, phone, headset, car, etc. Network access speeds, throughputs, reliabilities, are all important

issues.

OS view of the network may range from just a special form of file access to complex well-coordinated network operating systems.

Shared resources may include files, CPU cycles, RAM, printers, and other resources.

2013/09/29

38

Client-Server Computing A defined server provides services ( HW or SW ) to other systems

which serve as clients. ( Technically clients and servers are processes, not HW, and may co-exist on the same physical computer. )

A process may act as both client and server of either the same or different resources.

Served resources may include disk space, CPU cycles, time of day, IP name

information, graphical displays ( X Servers ), or other resources.

2013/09/29

39

General structure of a client-server system

Peer-to-Peer Computing There is no clear "leader" or overall organization.

Any computer or process on the network may provide services to any other which requests it.

May employ a central "directory" server for looking up the location of resources, or may use peer-to-peer searching to find resources.

E.g. Skype uses a central server to locate a desired peer, and then further communication is peer to peer.

2013/09/29

40

Peer-to-peer system with no centralized service

Cloud Computing Delivers computing, storage, and applications as a service over a network.

May contain thousands of physical computers, millions of virtual ones, and petabytes of total storage.

2013/09/29

41

Cloud Computing Types of cloud computing:

Public cloud - Available to anyone willing to pay for the service. Private cloud - Run by a company for internal use only. Hybrid cloud - A cloud with both public and private

components. Software as a Service - SaaS - Applications such as word

processors available via the Internet Platform as a Service - PaaS - A software stack available for

application use, such as a database server Infrastructure as a Service - IaaS - Servers or storage available

on the Internet, such as backup servers, photo storage, or file storage.

Service providers may provide more than one type of service

2013/09/29

42

Real-time Embedded Systems Embedded into devices

such as automobiles, climate control systems, process control, and even toasters and refrigerators.

May involve specialized chips, or generic CPUs applied to a particular task. ( Consider the current price of 80286 or even 8086 or 8088 chips, which are still plenty powerful enough for simple electronic devices such as kids toys. )

Process control devices require real-time ( interrupt driven ) OSes. Response time can be critical for many such devices.

2013/09/29

44

Open-Source Operating Systems Open-Source software

is published ( sometimes sold ) with the source code, so that anyone can see and optionally modify the code.

is often developed and maintained by a small army of loosely connected often unpaid programmers, each working towards the common good.

Critics argue that open-source SW can be buggy but proponents counter that bugs are found and fixed quickly,

since there are so many pairs of eyes inspecting all the code. Open-source operating systems are a good resource

for studying OS development, since students can examine the source code and even change it and re-compile the changes.

2013/09/29

45

Linux Developed by Linus Torvalds in Finland in 1991 as the first full

operating system developed by GNU.

Many different distributions of Linux have evolved from Linus's original,

including RedHat, SUSE, Fedora, Debian, Slackware, and Ubuntu, each geared toward a different group of end-users and operating environments.

To run Linux on a Windows system using VMware, follow these steps: Download the free "VMware Player" tool from

http://www.vmware.com/download/player and install it on your system

Choose a Linux version from among hundreds of virtual machine images at http://www.vmware.com/appliances

Boot the virtual machine within VMware Player.

2013/09/29

46

Utility The free software movement is gaining rapidly in popularity, leading

to thousands of ongoing projects involving untold numbers of programmers.

Sites such as http://freshmeat.net/ and http://distrowatch.com/ provide portals to many of these projects.

2013/09/29

47

BASIC COMMAND

2013/09/30

48

Basic file operations Start terminal

$ xterm Read the contents of file

$ cat shopping-list

Milk Apple Jam

$ more longfile

Copy file $ cp shopping-list last-week.list

Remove file $ rm old.data shopping.june2012

Move file (change the name) $ mv prog1.c first_program.c

49

2013/09/30

Basic directory operations Create and delete directory

$ mkdir /home/doly

$ mv first_program.c mycode

Lets try Move a file to a different directory Create a file under your home directory

50

2013/09/30

File Attribute Control file access through correct use of file and directory

permissions and ownership

$ id uid=505(doly) gid=20(staff) groups=20(staff),

$ ls l file.txta rwx rwx rwx User, Group, Other r: read, w: write, x: execute

$ touch test $ ls -l test

-rw-r--r-- 1 doly staff 0 3 21 11:17 test

51

2013/09/30

File operations Change ownership

Permit the execution privilege of a file for other $ chmod a+x test

ls -l test -rwxr-xr-x 1 doly staff 0 3 21 11:17 test Grant execution privilege for all users of test file

$ chmod g+w,g-r test

ls -l test -rwx-wxr-x 1 doly staff 0 3 21 11:17 test

52

2013/09/30

KERNEL DATA STRUCTURE

2013/09/30

53

Kernel Data Structures Lists

2013/09/30

54

Singly linked list

Doubly linked list

Circularly linked list

Kernel Data Structures Hash Function and Maps

Bitmaps A string of 1s and 0s used to keep track of the boolean state of a

collection of objects, such as the free state of blocks on a disk or pages in memory..

2013/09/30

55