2008 Oct Opensparc Slide Cast 09 Mw 1539018.HTML

7/24/2019 2008 Oct Opensparc Slide Cast 09 Mw 1539018.HTML

1/42

Creative Commons Attribution-Share 3.0 United States License1

www.opensparc.net

Creative Commons Attribution-Share 3.0 United States License

OpenSPARCSlide-Cast

This material is made available underCreative Commons Attribution-Share 3.0 United States License

In Twelve ChaptersPresented by penSPA!C desi"ners#developers# and pro"rammersto "uide users as they develop their ownpenSPA!C desi"ns andto assist pro$essors as they teach the ne%t"eneration
http://creativecommons.org/licenses/by-sa/3.0/us/http://www.opensparc.net/http://creativecommons.org/licenses/by-sa/3.0/us/http://creativecommons.org/licenses/by-sa/3.0/us/http://creativecommons.org/licenses/by-sa/3.0/us/http://creativecommons.org/licenses/by-sa/3.0/us/http://creativecommons.org/licenses/by-sa/3.0/us/http://www.opensparc.net/http://creativecommons.org/licenses/by-sa/3.0/us/


2/42


www.opensparc.net Creative Commons Attribution-Share 3.0 United States License

Maran Wilson

Software DeveloperSun Microsstems

Chapter Nine

HYPERVISOR ANDVIRTUALIATION
http://creativecommons.org/licenses/by-sa/3.0/us/http://www.opensparc.net/http://creativecommons.org/licenses/by-sa/3.0/us/http://creativecommons.org/licenses/by-sa/3.0/us/http://creativecommons.org/licenses/by-sa/3.0/us/http://www.opensparc.net/http://creativecommons.org/licenses/by-sa/3.0/us/


3/42

3www.opensparc.net Creative Commons Attribution-Share 3.0 United States License

S!n"! S#$tware%&ir'ware sta()

In the 'good old days' of sun4u, SW/FW was prettysimple. FW would ower on !pu, init registers, hand off to Solaris. Solaris re"oots, do a !lean fresh reset of the #W

SPARC hardware

Solaris

User

App

sun4u

User

App

OpenBoot (Reset/Config)

User

App
http://www.opensparc.net/http://creativecommons.org/licenses/by-sa/3.0/us/http://creativecommons.org/licenses/by-sa/3.0/us/http://www.opensparc.net/


4/424www.opensparc.net Creative Commons Attribution-Share 3.0 United States License

S!n"v Virt!ali*ati#n $#r SPARC plat$#r's

& sun4u to sun4$

sun4u code

Solaris(genunix)

CPU

(sun4v)

SPARC CPU

SPARC hypervisor

Solaris(genunix)

sun4v

interfaceCPUcode

peratin"

System

Plat$orm


5/42%www.opensparc.net Creative Commons Attribution-Share 3.0 United States License

Virt!al +a(hine appr#a(hes

&rue $irtualiation (mulate ma!hine ar!hite!ture down to register le$el

)an run any *S / software unmodified

erforman!e penalty

ara+$irtualiation efine a SW interfa!e to hyper$isor

-euires *S modifi!ation

ore effi!ient implementation + no emulation reuired

Sun's 0pproa!h 0 hy"rid + the pro!essors are designed for the sun4$ pri$ileged model,

"ut the hyper$isor impedan!e mat!hes anything not handled "y thehardware, and there are sun4$ 0Is implemented "y the hyper$isor.


6/42Creative Commons Attribution-Share 3.0 United States License

www.opensparc.net

Virt!al +a(hine $#r SPARC

Thin so$tware layer between S andplat$orm hardware

Para-virtualised S

'ypervisor ( sun)v inter$ace *irtualises machine '+ and isolates S $rom

re"ister-level

,elivered with plat$orm not S

ot itsel$ an SSPARC hardware

Hypervisor

Solaris

User

App

sun4v virtualmachine 0

stable interface sun4v

User

App

OpenBoot

Other OS

User

App

sun4v virtualmachine n

User

App...
http://creativecommons.org/licenses/by-sa/3.0/us/http://www.opensparc.net/http://www.opensparc.net/http://creativecommons.org/licenses/by-sa/3.0/us/


7/42Creative Commons Attribution-Share 3.0 United States License

www.opensparc.net

L#,i(al D#'ains

Hardware

Hypervisor

LDom 1

Solaris 10

CPU

Mem

LDom 2

Solaris 10

LDom 3

Solaris Next

Zone 1Zone Zone 2

Shared CPU,Memory, IO /

App

& artitioning !apa"ility )reate $irtual ma!hines

ea!h with su"+set ofresour!es

rote!tion Isolationusing #Wfirmware!om"ination

App

App

App

App

App

App

App

CPU CPU CPU

Mem Mem


8/42


www.opensparc.net

T#pi(s

& )6 !hanges

& emory management

& I/*

& Interrupts 7+!pu de$i!es

& ultiple omains

& 0dditional &opi!s (rror handling a!hine es!ription 8oot pro!ess


9/42


www.opensparc.net

asi( Prin(iples

& Ability to rebind virtualresources to physicalcomponents at any time

& 1inimal state held in'ypervisor to describe"uest S

& Never trust 2uest S

VirtualCPU

PhysicalCPU

(strand)

sun4v / APISlip-plane


10/42

Creative Commons Attribution-Share 3.0 United States License1:

www.opensparc.net

Le,a(. SPARC e/e(!ti#n '#de

Privile"ed1ode

User1ode

interrupts & errors

systemcalls

Retry

& *lder sun4u !hips ;6ltraS0-) I, II, III, I


11/42


www.opensparc.net

New SPARC E/e(!ti#n '#de

'yper-Privile"ed

1ode

Privile"ed1ode

User1ode

interrupts

& errors hypervisorcalls

Retry

systemcalls

Retry

Retry

interrupts& errors


12/42


www.opensparc.net

New SPARC E/e(!ti#n '#de

Virtual MachineEnvironment

'yper-Privile"ed

1ode

Privile"ed1ode

User1ode

interrupts& errors

hypervisor

calls

Retry

systemcalls

Retry

Retry

interrupts& errors


13/42


www.opensparc.net

Privile,ed '#de (#nstrained

& )lose deri$ati$e of lega!y pri$ileged mode, "ut> ?o a!!ess to diagnosti! registers ?o a!!ess to 6 !ontrol registers ?o a!!ess to interrupt !ontrol registers

?o a!!ess to I/*+6 !ontrol registers 0ll repla!ed "y #yper$isor 0I !alls

& 6ltraS0-)+ness remains with minor !hanges timer ti!@ registers softint registers et!. trap+le$els glo"al registers et!. register window spill/fill


14/42


www.opensparc.net

Translati#n hierar(h.

Physical AddressingHyper-privileged

LevelPartition IDN bit Context ID+ +

Real AddressingPrivileged

Level64bit Context ID+

Virtual AddressingUser Level

64bit

Virtual MachineEnvironment


15/42

Creative Commons Attribution-Share 3.0 United States License1%

www.opensparc.net

Translati#n 'ana,e'ent

& Auest *S defines a Bfault+statusC area of memory for ea!h$irtual )6 ;$)6= #yper$isor fills in info for ea!h 6 e7!eption

& 6ltraS0-) does not use page+ta"les traditionally a software loaded &D8

& #yper$isor 0Is to support dire!t software management of&D8 entries map, demap Simple guests li@e *8 !an use this


16/42


www.opensparc.net

Translati#n St#ra,e !$$ers

& Auest *S managed !a!he of translations stored inmemory Auest allo!ates memory for "uffer Auest pla!es translation mappings into "uffer when needed #yper$isor fet!hes from this !a!he into &D8s

& Auest spe!ifies $irtual +E real mappings #yper$isor translates real+Ephysi!al to load into &D8 &D8 holds $irtual +E physi!al mappings

& ultiple &S8s used simultaneously for multiple pagesies and !onte7ts


17/42


www.opensparc.net

Address spa(e (#ntr#l

Physical Address MapDom A: Real Map Dom B: Real Map

& #yper$isor limits a!!ess to memory and de$i!es ++!reating partitions ;logi!al domains=


18/42


19/42


www.opensparc.net

Ph.si(al I%O devi(es

& )I+(7press root !omple7 mapped into real addressspa!e of guest domain

& ire!t a!!ess to de$i!e registers *8 pro"es and !onfigures "us and de$i!es

& I/* 8ridge and I/* 6 !onfiguration $irtualied "yhyper$isor 0Is (nsures that I/* 6 translations are $alidated "y hyper$isor e$i!e interrupts are $irtualied for deli$ery


20/42

2:www.opensparc.net Creative Commons Attribution-Share 3.0 United States License

Dire(t I%O '#del

& For Solaris e7isting dri$ers!ontinue to wor@

'ypervisor

I/O

Bridge

PC rid"e,river

,evice,river

!ootComple%

/ 11U

Guest Domain

*irtual rid"e /4Hyper Privileged

Privileged

Hardware

PC-5%press

Domain owns PCIroot and tree


21/42


www.opensparc.net

Interr!pt and event deliver.

& e$i!e interrupts and error e$ents need a me!hanism toasyn!hronously !ause an e7!eption for a $irtual )6

& &ypi!ally also reuire some data to identify reason andsour!e and notifi!ation

& #ow to do this in an a"stra!t manner What if the $irtual )6 is not !urrently "ound to a physi!al

)6 )an't "lo!@ physi!al interrupt sour!e until $irtual )6 is

res!heduled


22/42


www.opensparc.net

Interr!pts 0 CPU '#nd#s

& eli$ered to pri$ileged mode $ia four in+memory FIF*ueues> !pu+mondo de$+mondo resuma"le error non+resuma"le

error

& 4+"yte entries !arry !ause information ;interrupt num"ers=& #ead and &ail offsets a$aila"le as )6 registers to

pri$ileged !ode &ail manipulated "y hyper$isor, head "y guest *S

For either ueue, head GH tail !auses trap

Head Tail


23/42


www.opensparc.net

1!e!e (#nstraints

& ust "e a power+of+2 num"er of entries, minimum of 2 (ntries always 4 "ytes in sie ;tail1=sie HH head defines full state

& ust "e aligned on a real address "oundary identi!al to

J sie esigned to ma@e hardware mondo deli$ery easier

& ay ha$e ueues defined for ea!h $irtual )6 de$Kmondo ueue must "e sied for all possi"le interrupt sour!es

de$Kmondo ueue may ne$er !ontain more than one entry for samesour!e

& #yper$isor 0I to send 48yte mondo to )6 ueue 6sed for )6 to )6 7+!alls Jueue may fill and sender's 0I !all fails


24/42


www.opensparc.net

L#,i(al D#'ainin, Te(hn#l#,.

&


25/42

2%www.opensparc.net Creative Commons Attribution-Share 3.0 United States License

&!nda'entals

& (a!h $irtual ma!hine should appear as an entirelyindependent ma!hine own @ernel, pat!hes, tuning parameters

own user a!!ounts, administrators

own dis@s own networ@ interfa!es, 0) I addresses

Start, stop and re"oot independently of ea!h other


26/42


H.pervis#r S!pp#rt

& #yper$isor software is responsi"le for maintainingseparation "etween domains 6sing e7tensions "uilt into a sun4$ )6

& 0lso pro$ides Dogi!al omain )hannels ;D)s= so thatdomains !an !ommuni!ate with ea!h other e!hanism "y whi!h domains !an "e $irtually networ@ed with

ea!h other, or pro$ide ser$i!es to ea!h other

& omain -oles> )ontrol, I/* and Ser$i!e


27/42


LD#' +ana,er

& *ne anager per ma!hine )an "e run in any domain, "ut only 1 domain at a time

&he B)ontrol omainC

)ontrols #yper$isor and all its Doms

& (7poses )DI to administrator

& aps Dogi!al omains to physi!al resour!es )onstraint engine

#euristi! "inding of Doms to resour!es0ssists with performan!e optimiation0ssists in e$en of failures / "la!@listing


28/42


www.opensparc.net

D.na'i( Re(#n$i,!rati#n

& #yper$isor has a"ility to dynami!ally shrin@ or growDoms upon demand

& Simply add/remo$e !pus, memory I/* 0"ility to !ope with this without re"ooting depends on guest

*S !apa"ilities Auest *S indi!ates its !apa"ilities to the domain manager

& *pportunity to impro$e utiliation "y "alan!ing resour!es"etween domains


29/42


Dire(t I%O

'ypervisor

I/O

Bridge

Nexus Driver/pci@B

,evice ,river/pci6/7lc68

PCI

Root

I/O MMU

Logical Domain

Virtual Nexus I/FHyperPrivileged

Privileged

Hardware

PC-5%press

Logical Domainowns PCI root andtree

& &raditional model (7isting dri$ers and de$i!es

!ontinue to wor@

App

App

App


30/42

Creative Commons Attribution-Share 3.0 United States License3:

www.opensparc.net

Virt!ali*ed I%O

Logical Domain A

'ypervisor

I/OBridge

Nexus Driver/pci@B

Device Driver/pci@B/qlc@6

PCIRoot

I/O MMU

Service Domain

Virtual Nexus I/FHyper

Privileged

Privileged

Hardware

PC

Domain Cannel

App

Virtual DeviceDriver

AppApp

App

Virtual DeviceService


31/42


Virt!al 2l#()3 Dis) devi(e 0 server

Logical Domain2

V-DiskDriver

Logical Domain3

V-DiskDriver

Logical Domain1

App

V-DiskDriver

Hyper-visor

ServiceDomain

Virtual SAN 1

Virtual SAN 2

I/O Bridge

FC-ALI/F

V-DiskBridge

V-DiskBridge

Device-

Driver

App

AppApp

App

AppApp

App

App

App


32/42


Red!ndan(.4 +!lti-path virt!al I%O

Logical Domain!

App

V-EtherDriver

Logical Domain3

V-EtherDriver

I/O Bridge

V-EtherSwitch

ServiceDomain2

GbEthernet I/F

Virtual LAN 1: 192.168.0/24

Virtual LAN 1b: 192.168.0/24

Device-Driver

I/O Bridge

ServiceDomain1

GbEthernet I/F

V-EtherSwitch

Device-Driver

V-EtherDriver

App

App

App

App

App

&


33/42


www.opensparc.net

Additi#nal T#pi(s

& (rror handling& a!hine es!riptions

& 8oot pro!ess


34/42


www.opensparc.net

Err#r deliver. phil#s#ph.

& #yper$isor handles and a"stra!ts underlying hardwareerrors

& 0ll errors logged on ser$i!e pro!essor

& Auest *Ss are told a"out impa!t of error ?o point in informing guest a"out !orre!ta"le errors

& Dega!y *S should "e a"le to run on new platform


35/42

Creative Commons Attribution-Share 3.0 United States License3%

www.opensparc.net

Err#r handlin,

& &wo simple !lassifi!iationsL Bafter handling the error, !an I ...C 1. -esume e7e!ution of what I was doing, or 2. )an't resume e7e!ution ... some poli!y to handle this

& Simplest error handlers> 1. -etry 2. ani!

& 2 in+memory ueues asso!iated with these types, similar tointerrupts

& Jueue entries !ontain error reports distilled "y hyper$isor

& #yper$isor !reates reports and attempts to !orre!t errorswhen possi"le


36/42


www.opensparc.net

Err#r handlin, a,ents

Hypervisor

non-resumable

error queue

resumable

error queue

device

mondo queue

Service Provider

I/F

CPU/mem/PIO

error handler

Virtual I/O

error handler

Direct I/O

error handlerFMA Agent

FMA AgentFMA error

report generator

Hardware

Error

Solaris

(Logical Domain)

Service Entity (Diagnosis Service Provider)

CPU/memory

error reports

I/O & device

error reports

Service error report


37/42


www.opensparc.net

+a(hine des(ripti#n

& #ow the *S Inside a $irtual ma!hine finds its resour!es& &ri$ial list of nodes that detail the !ontents of ea!h

domain )6s, 8lo!@s of memory, I/* de$i!es, I/* orts et!.

& ?odes are also inter+lin@ed to form a 0A to !on$eymore ad$an!ed information for guest *Ss that !are e.g. !a!he sharing, ?60 memory laten!ies et!.

& Mey reuirementsL


38/42


www.opensparc.net

Si'ple list #$ n#des

cpuid=0

root

cpus

memory

cpuid=1

cpuid=2

cpuid=3

MMU

memblkbase = 0x10000000size = 0x1000000


I/OVPCIid=0

FPU


39/42


www.opensparc.net

Als# arran,ed as a DA5

cpuid=0

root

cpus

memory

cpuid=1

cpuid=2

cpuid=3

MMU



I/OVPCIid=0

FPU


40/42

Creative Commons Attribution-Share 3.0 United States License4:www.opensparc.net

##t pr#(essPower on

Reset & configuration code

Hypervisor

OBP

Other specialistcodeufs/net boot

Solaris

exit/reboot

OBP

your bootloader

Your OS

exit/reboot


41/42

Creative Commons Attribution-Share 3.0 United States License41www.opensparc.net

S!''ar.

& Spe!ifi!ations !ode pu"lished> http>//www.openspar!.net http>//www.opensolaris.net

& BDegionC instru!tion le$el simulator a$aila"le to assist with

!ode de$elopment ro$ides le$el of !ode e7e!ution $isi"ility not possi"le on a!tual

hardware Sour!e !ode a$aila"le on http>//www.openspar!.net

& )onta!t alias> hyper$isorNsun.!om
http://creativecommons.org/licenses/by-sa/3.0/us/http://www.opensparc.net/http://www.opensolaris.net/http://www.opensparc.net/http://www.opensparc.net/http://www.opensolaris.net/http://www.opensparc.net/http://creativecommons.org/licenses/by-sa/3.0/us/


42/42

Creative Commons Attribution-Share 3.0 United States License42www.opensparc.net Creative Commons Attribution-Share 3.0 United States License

Than9s $or watchin" the

OpenSPARC Slide-Cast!Let us hear rom ou! "he OpenSPARC "eam#ould appreciate our eedbac$ on this in thehttp%#.OpenSPARC.net orum.

This material is made available underCreative Commons Attribution-Share 3.0 United States License
http://creativecommons.org/licenses/by-sa/3.0/us/http://www.opensparc.net/http://creativecommons.org/licenses/by-sa/3.0/us/http://creativecommons.org/licenses/by-sa/3.0/us/http://creativecommons.org/licenses/by-sa/3.0/us/http://creativecommons.org/licenses/by-sa/3.0/us/http://creativecommons.org/licenses/by-sa/3.0/us/http://www.opensparc.net/http://creativecommons.org/licenses/by-sa/3.0/us/

2008 Oct Opensparc Slide Cast 09 Mw 1539018.HTML

Documents

Transcript of 2008 Oct Opensparc Slide Cast 09 Mw 1539018.HTML