OSb: OSv on BitVisor (2)

45
OSb: OSv on BitVisor Yushi Omote University of Tsukuba Takahiro Shinagawa The University of Tokyo (2014/11/21 BitVisor Summit 3) Monday, November 24, 14

Transcript of OSb: OSv on BitVisor (2)

Page 1: OSb: OSv on BitVisor (2)

OSb: OSv on BitVisor

Yushi OmoteUniversity of Tsukuba

Takahiro ShinagawaThe University of Tokyo

(2014/11/21 BitVisor Summit 3)

Monday, November 24, 14

Page 2: OSb: OSv on BitVisor (2)

BitVisor

BitVisor

Hardware

OS

App App App

http://www.justis.as-1.co.jp

Monday, November 24, 14

Page 3: OSb: OSv on BitVisor (2)

OSv

VMM

Hardware

OSvApp

http://medical-care.feed.jp

Monday, November 24, 14

Page 4: OSb: OSv on BitVisor (2)

BitVisor

Monday, November 24, 14

Page 5: OSb: OSv on BitVisor (2)

BitVisor OSv

Monday, November 24, 14

Page 6: OSb: OSv on BitVisor (2)

BitVisor OSv

http://www.root.ne.jp/nishide/shs/

Monday, November 24, 14

Page 7: OSb: OSv on BitVisor (2)

OSv on BitVisor

BitVisor

Hardware

OSvApp

Monday, November 24, 14

Page 8: OSb: OSv on BitVisor (2)

OSv on BitVisor

BitVisor

Hardware

OSvApp

OSbMonday, November 24, 14

Page 9: OSb: OSv on BitVisor (2)

OSv on BitVisor

BitVisor

Hardware

OSvApp

OSb

What the fuckno Virtio !

Physical interfacesucks !

What the hell isPRO/1000 !

Monday, November 24, 14

Page 10: OSb: OSv on BitVisor (2)

OSv on BitVisor

BitVisor

Hardware

OSvApp

OSb

What the fuckno Virtio !

Physical interfacesucks !

What the hell isPRO/1000 !

You have no choice.

Virtio

Monday, November 24, 14

Page 11: OSb: OSv on BitVisor (2)

~ The Road to OSb ~

Monday, November 24, 14

Page 12: OSb: OSv on BitVisor (2)

• Boot Process

• Some Drivers

OSv Code Reading- MBR- Read local disk with INT13/42.- Load command line, boot loader, OSv kernel.- Get memory map with INT15/E820.- Setup segment descriptors/page tables.- Switch to 64-bit mode.- premain()- main()...

- Serial / VGA output.- SATA.- Virtio NIC/BLK/RNG/SCSI.- ACPI.- APIC....

OSv may run onphysical machine.(=BitVisor)

Monday, November 24, 14

Page 13: OSb: OSv on BitVisor (2)

Only one thing to fix

drivers/acpi.cc:

// Copy the root table list to dynamic memory if (!is_bitvisor()) { status = AcpiReallocateRootTable(); if (ACPI_FAILURE(status)) { acpi_e("AcpiReallocateRootTable failed: %s\n", AcpiFormatException(status)); return; } }

Triple faultSkip on BitVisor

Monday, November 24, 14

Page 14: OSb: OSv on BitVisor (2)

Hello world !

Monday, November 24, 14

Page 15: OSb: OSv on BitVisor (2)

OSb

Whenever updating OSv images,you update the entire Physical Disk.

Troublesome...

http://www.dreamstime.comhttp://a.fsdn.com/sd/articles/14/11/12/1946208-1.jpg

On BitVisor,

(especially for development)

Monday, November 24, 14

Page 16: OSb: OSv on BitVisor (2)

Network-boot of OSb

./scripts/run.py

Monday, November 24, 14

Page 17: OSb: OSv on BitVisor (2)

Network-boot of OSb

Hardware

./scripts/run.py

Monday, November 24, 14

Page 18: OSb: OSv on BitVisor (2)

Network-boot of OSb

Hardware

./scripts/run.py

ImageUpdate

Monday, November 24, 14

Page 19: OSb: OSv on BitVisor (2)

Network-boot of OSb

Hardware

./scripts/run.py

ImageUpdate

IPMI

Monday, November 24, 14

Page 20: OSb: OSv on BitVisor (2)

Network-boot of OSb

Hardware

./scripts/run.py

ImageUpdate

IPMIPXE Boot

Monday, November 24, 14

Page 21: OSb: OSv on BitVisor (2)

Network-boot of OSb

Hardware

./scripts/run.py

ImageUpdate

IPMIPXE Boot

BitVisor

Monday, November 24, 14

Page 22: OSb: OSv on BitVisor (2)

Network-boot of OSb

Hardware

./scripts/run.py

ImageUpdate

IPMIPXE Boot

BitVisor

OSvNetwork Boot

Monday, November 24, 14

Page 23: OSb: OSv on BitVisor (2)

Network-boot of OSb

Hardware

./scripts/run.py

ImageUpdate

IPMIPXE Boot

BitVisor

OSvNetwork Boot

App

Monday, November 24, 14

Page 24: OSb: OSv on BitVisor (2)

Network-boot of OSb

BitVisor

Hardware

OSvApp

Network Boot

IPMI

./scripts/run.py

TTY

ImageUpdate

PXE Boot

Monday, November 24, 14

Page 25: OSb: OSv on BitVisor (2)

Performance?

0

6.50

13.00

19.50

26.00

OSb Linux (Host, Bare-metal)

Elap

sed

Tim

e (s

ec)

for (12-billion times) val++;

Monday, November 24, 14

Page 26: OSb: OSv on BitVisor (2)

Virtio NIC

BitVisor

Hardware

OSvApp

PRO/1000...

No PRO/1000...Virtio NIC please...

Monday, November 24, 14

Page 27: OSb: OSv on BitVisor (2)

Virtio NIC

BitVisor

Hardware

OSvApp

Virtio NIC

PRO/1000...

Implementing Virtio

No PRO/1000...Virtio NIC please...

Monday, November 24, 14

Page 28: OSb: OSv on BitVisor (2)

How to implement?

BitVisor

Hardware

OSvApp

Virtio NIC

Finally, OurVirtualization?

Monday, November 24, 14

Page 29: OSb: OSv on BitVisor (2)

How to implement?

BitVisor

Hardware

OSvApp

Virtio NIC

Finally, OurVirtualization?

No,BitVisor’s Way!

Monday, November 24, 14

Page 30: OSb: OSv on BitVisor (2)

Para Pass-through Virtio

BitVisor

Hardware

Virtio Ring

PRO/1000 Ring

OSvApp

MSI-X etc.

Intercept only interesting I/Os!

PCI Functions

Pass-through!

Monday, November 24, 14

Page 31: OSb: OSv on BitVisor (2)

Faking PCI IDs 0x1AF4(Virtio Device)

0x1000(Virtio NIC)

0 (Legacy Virtio)

1 (Virtio NIC)

Monday, November 24, 14

Page 32: OSb: OSv on BitVisor (2)

Faking PCI BARsBAR0: IO spacefor Virtio = PRO/1000 IO space

Everything elsePass-through(e.g. MSI-X)

Monday, November 24, 14

Page 33: OSb: OSv on BitVisor (2)

Virtio NIC Operations

• Virtio Ring

• Packet Transmission

• Packet Reception

Monday, November 24, 14

Page 34: OSb: OSv on BitVisor (2)

Virtio RingGuest selects rings: * Transmission Ring * Reception Ring

BAR0(IO)

Memory space

Req. Req.Req.

Virtio Ring (Available + Used)

BitVisor emulatesthese registers

Monday, November 24, 14

Page 35: OSb: OSv on BitVisor (2)

Packet Transmission

OSv(Driver)

BitVisor(Virtio NIC)

Req.

PRO/1000

Req.

Transmission Ring

Req.

Req. Req. Req.

Notify!(PIO)

Req.

Req.

Reuse

Transmission Ring

VMExit

Monday, November 24, 14

Page 36: OSb: OSv on BitVisor (2)

Packet Reception

OSv(Driver)

BitVisor(Virtio NIC)

Req.

PRO/1000

Req.

Reception Ring

Req.

Req. Req. Req.

Interrupt

Req.

Req.

Reuse

Reception Ring

VMExit

Monday, November 24, 14

Page 37: OSb: OSv on BitVisor (2)

Netperf TCP_STREAM

0

250.00

500.00

750.00

1000.00

OSb Linux (Host, Bare-metal)

941.39

816.78

Thr

ough

put

(Mbp

s)

Monday, November 24, 14

Page 38: OSb: OSv on BitVisor (2)

VMExits are Costly...

OSv(Driver)

BitVisor(Virtio NIC)

PRO/1000

Interrupt

VMExit

Notify!(PIO)

VMExit

Guestmode

Hostmode

SwitchingOverhead

Monday, November 24, 14

Page 39: OSb: OSv on BitVisor (2)

Exitless Virtio with Dedicated Core

Req. Req.Req.Transmission

Ring

Req. Req.Req.Reception

Ring

Req. Req.Req.

Req. Req.Req.

Interrupt (IPI)

AlwaysGuestMode

AlwaysHostMode

Polling

ELVIS [Har’El et al. ATC’13]

Monday, November 24, 14

Page 40: OSb: OSv on BitVisor (2)

Implementation Summary

• Core Concealing

• Modifying ACPI MADT Table

• Notification PIO Pass-through

• Pass-through to ineffective PIO

• Interrupt

• Get vector # from MSI-X table and Send IPI

• (Or ask PRO/1000 to trigger)

Monday, November 24, 14

Page 41: OSb: OSv on BitVisor (2)

Netperf TCP_STREAM(Exitless Virtio)

0

237.50

475.00

712.50

950.00

OSb Linux (Host, Bare-metal)

941.39921.83

Thr

ough

put

(Mbp

s)

Monday, November 24, 14

Page 42: OSb: OSv on BitVisor (2)

Ping(Exitless Virtio)

0

0.04

0.08

0.12

0.16

OSb Linux (Host, Bare-metal)

0.16

0.12

Late

ncy

(mse

c)

Monday, November 24, 14

Page 43: OSb: OSv on BitVisor (2)

Still Under Optimization

• TODOs

• Interrupt Moderation

• Zero Copy/VMDq

• Advanced PRO/1000 Descriptor

• TCP Offloading

Monday, November 24, 14

Page 44: OSb: OSv on BitVisor (2)

Summary &Future Work

• Summary

• Running OSv on BitVisor

• Virtio NIC + Optimization

• Future work

• Further Optimization & Evaluation

• Other Virtio Devices (BLK, RNG...)

Monday, November 24, 14

Page 45: OSb: OSv on BitVisor (2)

Thank you !

http://www.root.ne.jp/nishide/shs/

Monday, November 24, 14