Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos...
Transcript of Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos...
![Page 1: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/1.jpg)
Ankit SinglaData center architecture
![Page 2: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/2.jpg)
2
A server rack
A rack of servers
A top-of-rack switch
![Page 3: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/3.jpg)
3
Lots of racks
How to network the racks?
”
![Page 4: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/4.jpg)
“Big switch” approach
4
…
![Page 5: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/5.jpg)
“Big switch” approach
5
…
![Page 6: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/6.jpg)
Jupiter Rising: A Decade of Clos Topologies and
Centralized Control in Google’s Datacenter Network
Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon,Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost,Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat
Google, Inc.j it i @ l
ACM SIGCOMM, 2015
512 ports x 1G
40 servers x 1G
4 ports x 1G
610:1 “oversubscription”
![Page 7: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/7.jpg)
“Big switch” approach
7
$$$$$$$$$$ $$$$$$$$$$
…
![Page 8: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/8.jpg)
$$$ $$$
Alternative: tree network
8
![Page 9: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/9.jpg)
9
Connect many cheap, identical switches?
Goals: high capacity, low latency
![Page 10: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/10.jpg)
If you know your application …
10
![Page 11: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/11.jpg)
… design for it
11
NERSC
![Page 12: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/12.jpg)
But, other apps may not work well …
12
![Page 13: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/13.jpg)
13
What’s so hard about this?6-port switches
4-port switches
�
���
�����
�����
�����
������
������
� � � �� �� �� �� �� ��
������������������������
������ �� ��������
![Page 14: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/14.jpg)
14
What’s so hard about this?
![Page 15: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/15.jpg)
15
Clos networks
Use small, cheap elements to build large networks!!
![Page 16: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/16.jpg)
Fat-tree
![Page 17: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/17.jpg)
17
Fat-tree network
Pod 0
10.0.2.1
10.0.1.1
Pod 1 Pod 3Pod 210.2.0.2 10.2.0.3
10.2.0.1
10.4.1.1 10.4.1.2 10.4.2.1 10.4.2.2
10.2.2.1
10.0.1.2
ACM SIGCOMM, 2008
A Scalable, Commodity Data Center Network Architecture
Mohammad Al-Fares Alexander Loukissas Amin Vahdat
Servers
Core
![Page 18: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/18.jpg)
18
Fat-tree network
![Page 19: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/19.jpg)
19
Jupiter Rising: A Decade of Clos Topologies and
Centralized Control in Google’s Datacenter Network
Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon,Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost,Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat
Google, Inc.j it i @ l
ACM SIGCOMM, 2015
![Page 20: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/20.jpg)
[Image: Robert Harker]
![Page 21: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/21.jpg)
[Image: Robert Harker]
![Page 22: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/22.jpg)
22
Jupiter Rising: A Decade of Clos Topologies and
Centralized Control in Google’s Datacenter Network
Arjun Singh, Joon Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon,Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano, Anand Kanagala, Jeff Provost,Jason Simmons, Eiichi Tanda, Jim Wanderer, Urs Hölzle, Stephen Stuart, and Amin Vahdat
Google, Inc.j it i @ l
ACM SIGCOMM, 2015
![Page 23: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/23.jpg)
23
Cabling[Robert Harker
![Page 24: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/24.jpg)
24
DC architecture
How to network the racks?
But can we do better?
![Page 25: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/25.jpg)
25
Other topologies besides fat-trees?
6-port switches
4-port switches
�
���
�����
����������������������
� � � �� �� �� �� �� ��������������������������
������ �� ��������
Surprise: picking randomly from this space works better than fat-trees!
Deterministic expander constructions can also achieve these gains
![Page 26: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/26.jpg)
But do we need full bandwidth everywhere at all times?
![Page 27: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/27.jpg)
25% is the high mark for overall utilization (?) ”
2012.
27
![Page 28: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/28.jpg)
hotspots
Skew in network traffic
![Page 29: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/29.jpg)
29
How does this observation help?
“Unfortunately, given current data center network architectures, the only way to provision required bandwidth bandwidth between dynamically changing sets of nodes is to build a non-blocking switch fabric at the scale of an entire data center”
.
![Page 30: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/30.jpg)
Set up network connections on the fly?
Servers
ToR
switch
Aggregate
switch
Core
switch
Electrical
Network
Optical
Network
Reconfigurable
optical paths
Array of Micromirrors
Diffracted beam Towards destinationReceived beam
Input beam LasersDMDs
Photodetectors
Mirror assembly Reflected beam
TX
RX
Rack%1%Rack%N%Rack%r%
Steerable%%FSOs%
Ceiling%mirror%
ToR%switch%
FireFly%Controller%
Traffic%Pa=erns%
Rule%change%
FSO%reconf%
Can statically wired networks compete with this?
![Page 31: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/31.jpg)
31
DC architecture @ SIGCOMM 2017RotorNet: A Scalable, Low-complexity,
Optical Datacenter Network
William M. Mellette, Rob McGuinness, Arjun Roy, Alex Forencich,
George Papen, Alex C.Snoeren, and George Porter
University of California, San Diego
A
T
a
l
e
o
f
T
w
o
T
o
p
o
l
o
g
i
e
s
:
E
x
p
l
o
r
i
n
g
C
o
n
v
e
r
t
i
b
l
e
D
a
t
a
C
e
n
t
e
r
N
e
t
w
o
r
k
A
r
c
h
i
t
e
c
t
u
r
e
s
w
i
t
h
F
l
a
t
-
t
r
e
e
Yiting Xia, Xiaoye Steven Sun, Simbarashe Dzinamarira,
Dingming Wu, Xin Sunny Huang, T. S. Eugene Ng
Rice University
Beyond fat-trees without antennae, mirrors, and disco-balls
Simon KassingETH Zürich
Asaf ValadarskyHebrew University of Jerusalem
Gal ShahafHebrew University of Jerusalem
Michael SchapiraHebrew University of Jerusalem
Ankit SinglaETH Zürich
![Page 32: Data center architecture - acm sigcomm · 2017-10-27 · Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network Arjun Singh, Joon Ong,](https://reader030.fdocuments.us/reader030/viewer/2022040223/5e5125d78341bc50d16d5c0d/html5/thumbnails/32.jpg)