Evolution of Distributed Operating Systems and Virtualization · • New H/W features are not...
Transcript of Evolution of Distributed Operating Systems and Virtualization · • New H/W features are not...
Evolution of Distributed Operating Systems and
Virtualization
21 November 2009
Jun Nakajima
Principal Engineer
2
Agenda
• History
• Microkernel based Distributed Operating Systems
• Current Generation and Virtualization
• Next Generation (My View)
3
17 ½ Years Ago in Seattle, Washington, US
4
Some of them are still there
5
Distributed UNIX* SVR4 on Chorus
Workshop Proceedings “Micro-kernels and Other Kernel Architectures”,
April 27 – 28, 1992, Seattle, Washington
Process Manger
Object Manger (File)
Stream Manger
IPC Key Manger
6
Microkernel-Based Kernels
• Failed as foundation of distributed operating systems
− Required significant “restructuring” of the OS
• Maintenance /compatibility issues with original OS
− Did not match with the abstractions implemented by OS
• Design/logic changes
• Too simple or generic
− More complexity
• Distributed systems are more complex, hard to debug, APIs keep changing…
• But provided innovations toward modular, optimized, well-
structured OS
− Hybrid kernel
− Nanokernel
• Used by various embedded/real-time OS todayDistributing single OS in components was not a good idea
7
BTW: More Deterministic Latency on Micro-kernel Based OS
Workshop Proceedings “Micro-kernels and Other Kernel Architectures”,
April 27 – 28, 1992, Seattle, Washington
Hypervisor-based Real-time systems can be good
choice
8
Agenda
• History
• Microkernel based Distributed Operating Systems
• Current Generation and Virtualization
• Next Generation (My View)
Virtual Machine Monitor (VMM)
MemoryProcessors I/O Devices
Storage
Network
PhysicalPlatformResources
VM0…
VirtualMachines(VMs)
OS
Apps
VM1
OS
Apps
VM2
OS
Apps
VMn
OS
Apps
VMM is similar to operating system; it monitors
VMs instead of processes
10
Virtualization-Based Distributed OS
• Virtualization on VMM (Virtual Machine Monitor) maintains
existing raw and machine interfaces
− No need to change or restructure the OS
− Well-established interface/behavior based on H/W
− Para-virtualization changes the interface, but the changes are typically
mechanical.
• No design/logic changes
• Virtualization-based approach is ideal for implementing
distributed OS
− Scalability (next slide)
− Reliability (in “Extended topics on virtualization” Class)
− H/W virtualization features
Hardware is fully ready for Virtualization-Based
Distributed Operating System
11
Achieving Scalability Using VMM
• Scalability using single VM
− Scalability within VM
• Increase VCPUs (virtual CPU hot-plug), amount of memory (virtual memory hot-
plug), I/O capabilities (hot-plug, direct I/O, SR-IOV)
− Scalability using VMs
• Add more VMs
• Scalability using multiple VMs
− Use live migration
Physical Host #1
VM Monitor (VMM)
Physical Devices
VM1
Guest OS 1
App #2
Virtual
Devices
VM1
App #2
Virtual
Devices
VM2
Guest OS 3
App #3
Virtual
Devices
VM2
Guest OS2
App #3
Virtual
Devices
VM0
Guest OS 0
App #1
Virtual
Devices
VM0
Guest OS0
App #1
Virtual
Devices
Guest OS1
Physical Host #2
VM Monitor (VMM)
Physical Devices
VM2
App #3
Virtual
Devices
Guest OS2
VM1
App #2
Virtual
Devices
Guest OS1
Physical Host #m
VM Monitor (VMM)
Physical Devices
VM1
App #2
Virtual
Devices
VM2
App #3
Virtual
Devices
Guest OS2 Guest OS1
Physical Host #x
VM Monitor (VMM)
Physical Devices
VM1
App #2
Virtual
Devices
VM2
App #3
Virtual
Devices
Guest OS2 Guest OS1
VM1
App #2
Virtual
Devices
Guest OS1
VM1
App #2
Virtual
Devices
Guest OS1
12
Current Generation and Status
• VMM was not intended to be a distributed OS, but some
features are emerging
− Resource balancing using live migration
− Live migration is not available across VMMs from different venders
• Probably it will not happen in the short term
• Management tools manage resources and VMMs react
− VM configuration
− VM creation/termination
− VM live migration
• Management tools are semi-automatic
− Better than “by hand”
− Not scalable (with e.g. >100)
Current Generation is Incomplete as Distributed
Operating System
13
Agenda
• History
• Microkernel based Distributed Operating Systems
• Current Generation and Virtualization
• Next Generation (My View)
14
Issues with Current Generation
Physical Host #1
VM Monitor (VMM)
Physical Devices
VM1
Guest OS 1
App #2
Virtual
Devices
VM1
App #2
Virtual
Devices
VM2
Guest OS 3
App #3
Virtual
Devices
VM2
Guest OS2
App #3
Virtual
Devices
VM0
Guest OS 0
App #1
Virtual
Devices
VM0
Guest OS0
App #1
Virtual
Devices
Guest OS1
Physical Host #2
VM Monitor (VMM)
Physical Devices
VM1
App #2
Virtual
Devices
VM2
App #3
Virtual
Devices
Guest OS2 Guest OS1
Physical Host #m
VM Monitor (VMM)
Physical Devices
VM1
App #2
Virtual
Devices
VM2
App #3
Virtual
Devices
Guest OS2 Guest OS1
Physical Host #x
VM Monitor (VMM)
Physical Devices
VM1
App #2
Virtual
Devices
VM2
App #3
Virtual
Devices
Guest OS2 Guest OS1
H/W Abstraction Layer
• Minimal H/W abstraction layer to VMs
− Advanced H/W virtualization features are not effectively advertized
• Direct I/O (e.g. VT-d), SR-IOV, HAP (Hardware Assisted Paging, e.g. EPT) or not
− Least common features advertized for live migration
• New H/W features are not utilized in VMs effectively
• VM configuration/placement is ad hoc (e.g. hand)
• Number of VMs and machines would increase in the future
− Multi-cores, Cloud Computing
− More frequent resource adjustment/management
• >10,000 VMs
We will need “real” Virtualization-Based Distributed
Operating System
15
Next Generation: Cloud Operating Systems
• More scalability using systematic and automatic resource management
− View the whole system
− Monitor each VMM and VMs
• Host level: CPU features, # of sockets, # of cores, CPU utilization, # of interrupts, amount memory available, power consumption, I/O throughput, devices available, H/W virtualization features,
• Per VM: VCPU features, # of sockets, # of cores, # of VM exits, # of interrupts, power consumption, I/O throughput and flows, devices available,
− VM creation/placement based on profile/requirements and policy/rules
• Priority
• Type of VM, e.g. large and I/O intensive, HPC, desktop, NUMA,
• Power management – E.g. suspend if idle for a specific time
• Failover, etc.
− Prediction
16
Next Generation: Cloud Operating Systems (cont.)
• Utilize new/advanced H/W features in VMs as much as possible
for cloud computing
− Based on VM profile/requirements
• Performance, power, workload characteristics, security, etc.
− Minimize virtualization overheads
• # of VMs can be very large
• More advance topics
− Grid computing, HPC cloud
• Parallel computing, memory intensive workload, high throughput
17
Conclusions
• Virtualization technologies allow to implement “real” Distributed
Operating System for next generation
− Scalable/reliable cloud operating systems
− High-performance/Grid computing
• Hardware virtualization features are ready for solving
challenging problems
• Time for you to work in virtualization!