Operating Systems Meet Fault Tolerance

Operating Systems Meet Fault Tolerance

Bjorn Dobel

06.08.2013

Bjorn Dobel () OS Resilience 06.08.2013 1 / 58

Introduction

“If there’s more than one possible outcome of a job or task,

and one of those outcome will result in disaster or an

undesirable consequence, then somebody will do it that way.”

(Edward Murphy jr.)


Introduction

Outline

Murphy and the OS: Is it really that bad?

Fault-Tolerant Operating SystemsMinix3CuriOSL4ReAnimator

Dealing with Hardware ErrorsTransparent replication as an OS service


Introduction

Why Things go Wrong

Programming in C:

This pointer is certainly never going to be NULL!

Layering vs. responsibility:

Of course, someone in the higher layers will already have checkedthis return value.

Concurrency:

This struct is shared between an IRQ handler and a kernel thread.But they will never execute in parallel.

Hardware interaction:But the device spec said, this was not allowed to happen!

Hypocrisy:

I’m a cool OS hacker. I won’t make mistakes, so I don’t need totest my code!


Introduction

A Classic Study

A. Chou et al.: An empirical study of operating system errors, SOSP 2001

Automated software error detection (today: http://www.coverity.com)

Target: Linux (1.0 - 2.4)Where are the errors?How are they distributed?How long do they survive?Du bugs cluster in certain locations?


Introduction

Revalidation of Chou’s Results

N. Palix et al.: Faults in Linux: Ten years later, ASPLOS 2011

10 years of work on tools to decrease error counts - has it worked?

Repeated Chou’s analysis until Linux 2.6.34


Introduction

Linux: Lines of Code


Introduction

Fault Rate per Subdirectory (2001)


Introduction

Fault Rate per Subdirectory (2011)


Introduction

Bug Lifetimes (2011)


Introduction

Bug Distribution


Introduction

Break

Faults are an issue.

Hardware-related stu↵ is worst.

Now what can the OS do about it?


Introduction

Minix3 – A Fault-tolerant OS

Userprocesses User Processes

Server Processes

Device Processes Disk TTY Net Printer Other

File PM Reinc ... Other

Shell Make User ... Other

Kernel

Kernel Clock Task System Task


Introduction

Minix3: Fault Tolerance

Address Space IsolationApplications only access private memoryFaults do not spread to other components

User-level OS servicesPrinciple of Least PrivilegeFine-grain control over resource access

e.g., DMA only for specific drivers

Small componentsEasy to replace (micro-reboot)


Introduction

Minix3: Fault Detection

Fault model: transient errors caused by software bugs

Fix: Component restart

Reincarnation server monitors componentsProgram termination (crash)CPU exception (div by 0)Heartbeat messages

Users may also indicate that something is wrong


Introduction

Repair

Restarting a component is insu�cient:Applications may depend on restarted componentAfter restart, component state is lost

Minix3: explicit mechanismsReincarnation server signals applications about restartApplications store state at data store serverIn any case: program interaction needed

Restarted app: store/recover stateUser apps: recover server connection


Introduction

Break

Minix3 fault tolerance:Architectural IsolationExplicit monitoring and notifications

Other approaches:CuriOS: smart session state handlingL4ReAnimator: semi-transparent restart in a capability-based system


Introduction

CuriOS: Servers and Sessions

State recovery is trickyMinix3: Data Store for application dataBut: applications interact

Servers store session-specific stateServer restart requires potential rollback for every participant

ServerState

ClientAState

ClientBStateServer

Client A

Client B


Introduction

CuriOS: Server State Regions

CuriOS kernel (CuiK) manages dedicated session memory: Server StateRegions

SSRs are managed by the kernel and attached to a client-server connection

ServerState

Server

Client A

Client State A

Client B

Client State B


Introduction

CuriOS: Protecting Sessions

SSR gets mapped only when a client actually invokes the server

Solves another problem: failure while handling A’s request will never corruptB’s session state

ServerState

Server

Client A

Client State A

Client B

Client State B


Introduction




ServerState

Server

Client A

Client State A

Client B

Client State B

c

a

l

l

(

)


Introduction




ServerState

Server

Client A

Client State A

Client B

Client State B

ClientAState

c

a

l

l

(

)


Introduction




ServerState

Server

Client A

Client State A

Client B

Client State B

r

e

p

l

y

(

)


Introduction




ServerState

Server

Client A

Client State A

Client B

Client State Bc

a

l

l

(

)


Introduction




ServerState

Server

Client A

Client State A

Client B

Client State BClientBState

c

a

l

l

(

)


Introduction




ServerState

Server

Client A

Client State A

Client B

Client State Br

e

p

l

y

(

)


Introduction

CuriOS: Transparent Restart

CuriOS is a Single-Address-Space OS:Every application runs on the same page table (with modified access rights)

OS A A B BShared Mem

OS A A B BB Running

OS A A B BA Running

OS A A B BVirt. Memory


Introduction

Transparent Restart

Single Address SpaceEach object has unique addressIdentical in all programsServer := C++ object

RestartReplace old C++ object with new oneReuse previous memory locationReferences in other applications remain validOS blocks access during restart


Introduction

L4ReAnimator: Restart on L4Re

L4Re ApplicationsLoader component: nedDetects application termination: parent signalRestart: re-execute Lua init script (or parts of it)

Problem after restart: capabilitiesNo single component knows everyone owning a capability to an objectMinix3 signals won’t work


Introduction

L4Re: Session Creation

Client Server

Loader

SessionCreationCapability


Introduction


Client Server

Loader

(1)create


Introduction


Client Server

Loader

(2) Mappedduring startup


Introduction


Client Server

Loader

(3) factory.create()


Introduction


Client Server

Loader

Session Capability


Introduction


Client Server

Loader

(4) create


Introduction


Client Server

Loader

(5) use


Introduction

L4Re: Server Crash

Client Server

Loader

Session Capability

(6)exit

Kernel destroys mem-ory, server objects(channels...)


Introduction

L4Re: Server Crash

Client Server

Loader

Session Capability

(7)restart


Introduction

L4Re: Restarted Server

Client Server

Loader

Session Capability


Introduction


Client Server

Loader

Session Capability

use


Introduction


Client Server

Loader

Session Capability

use

Error!


Introduction

L4ReAnimator

Only the application itself can detect that a capability vanished

Kernel raises Capability fault

Application needs to re-obtain the capability: execute capability fault handler

Capfault handler: application-specificCreate new communication channelRestore session state

Programming model:Capfault handler provided by server implementorHandling transparent for application developerSemi-transparency


Introduction

L4ReAnimator: Cleanup

Some channels have resources attached (e.g., frame bu↵er for graphicalconsole)

Resource may come from a di↵erent resource (e.g., frame bu↵er frommemory manager)

Resources remain intact (stale) upon crash

Client ends up using old version of the resource

Requires additional app-specific knowledge

Unmap handler


Introduction

Summary

L4ReAnimatorCapfault: Clients detect server restarts lazilyCapfault Handler: application-specific knowledge on how to regain access tothe serverUnmap handler: clean up old resources after restart

All these frameworks only deal with software errors.

What about hardware faults? ! Next session!


Operating Systems Meet Fault Tolerance

Education

Transcript of Operating Systems Meet Fault Tolerance