Post on 26-Aug-2014
description
Docker + InfiniBand@syoyo
Docker Meetup in Tokyo #1(Feb 12, 2014)
Thursday, February 13, 14
Why InfiniBand?
• Faster networking
• 3.2 GB/s on QDR
• Low laterncy
• ~3 us
• de-facto inter-connect standard in HPC environment
Thursday, February 13, 14
Crehan Research
Thursday, February 13, 14
My contribution (1/2)
Thursday, February 13, 14
My contribution (2/2)
• OFED userland port for OpenIndiana(Solaris)
• https://github.com/syoyo/oi-build/tree/ofed-1.5.4.1/components/open-fabrics
Thursday, February 13, 14
Existing virtualization technique
• SR-IOV
• HW dependent.
• Need driver/firmware support
• PCI-passthrough
• Only one guest can access
• ZeroVM
• NaCl-based application isolation
Thursday, February 13, 14
LXC
• Lightweight virtualization
• Direct-access to PCI-ex devices
Thursday, February 13, 14
Existing research
• Performance Evaluation of Container-based Virtualization for High Performance Computing Environments
• Xavier, M.G. et. al. PDP2013
• http://marceloneves.org/papers/pdp2013-containers.pdf
Thursday, February 13, 14
• Simple, easy but powerful!
Thursday, February 13, 14
User A User B User C
Memory
CPU
OS
(*) http://commons.wikimedia.org/wiki/Smiley
(*)
Thursday, February 13, 14
Container A
Memory
CPU
Container B Container C
OS
Memory
CPU
Memory
CPU
Thursday, February 13, 14
User A User B User C
/opt/gcc-4.4
/opt/gcc-4.5
/opt/gcc-4.6
Thursday, February 13, 14
Container A
gcc4.4
Container B Container C
OS
gcc4.5 gcc4.6
Thursday, February 13, 14
Container(CentOS6.5)
CentOS 6.5
IB switch
ConnectX-2
CentOS 6.5
ConnectX-2
192.168.11.16
192.168.11.8
Thursday, February 13, 14
• Privilege : on
• bind /dev/infiniband
$ sudo docker run -privileged \ -v /dev/infiniband:/dev/infiniband \ -i -t centos6-ib /bin/bash
Thursday, February 13, 14
Thursday, February 13, 14
# ib_read_bw 192.168.11.16…----------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] 65536 1000 3008.83 3008.80 0.048141----------------------------------------------------
# ib_write_bw 192.168.11.16…----------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] 65536 5000 3066.07 3066.05 0.049057---------------------------------------------------------------------------------------
# ib_read_lat 192.168.11.16…----------------------------------- #bytes #iterations t_min[usec] t_max[usec] t_typical[usec] 2 1000 1.66 20.13 2.65 ---------------------------------------------------------------------------------------
# ib_write_lat 192.168.11.16…----------------------------------- #bytes #iterations t_min[usec] t_max[usec] t_typical[usec] 2 1000 0.80 7.12 0.86 ---------------------------------------------------------------------------------------
LXC Container
Thursday, February 13, 14
$ ib_read_bw 192.168.11.16…----------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] 65536 1000 3015.43 3011.54 0.048185---------------------------------------------------------------------------------------
$ ib_write_bw 192.168.11.16…----------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] 65536 5000 3065.79 3065.63 0.049050--------------------------------------------------------------------------------
$ ib_read_lat 192.168.11.16…----------------------------------- #bytes #iterations t_min[usec] t_max[usec] t_typical[usec] 2 1000 1.68 39.48 2.67 ---------------------------------------------------------------------------------------
$ ib_write_lat 192.168.11.16…----------------------------------- #bytes #iterations t_min[usec] t_max[usec] t_typical[usec] 2 1000 0.80 4.70 0.86 ---------------------------------------------------------------------------------------
Native
Thursday, February 13, 14
Details
• http://qiita.com/syoyo/items/bea48de8d7c6d8c73435
• in Japanese
Thursday, February 13, 14
Summary
• Docker + InfiniBand works!
• No virtualization overhead!
Thursday, February 13, 14
Future work(1/3)
• Per-container(per-user) access control for IB device
• e.g. IB port0 -> User A, IB port1 -> User B
• Security concern
• Other PCI-ex devices
• GPU
• PCI-SSD
• etc.
Thursday, February 13, 14
Future work(2/3)
• Virtualized, highly-efficient cloud platform for rendering task.
Thursday, February 13, 14
Render A Render B Render C Render D
Render E
InfiniBand
etcd
Future work(3/3)
Thursday, February 13, 14
• Thank you!
Thursday, February 13, 14