Ove

rvie

w o

f Su

perc

ompu

ter S

yste

ms

Supe

rcom

putin

g D

ivis

ion

Info

rmat

ion

Tech

nolo

gy C

ente

rTh

e U

nive

rsity

of T

okyo

2

Sup

erco

mpu

ters

at I

TC, U

. of T

okyo

Tota

l Use

rs >

2,0

002

Total

Pea

k per

forma

nce

: 140

TFL

OPS

Total

numb

er o

f nod

es: 9

52To

tal m

emor

y: 3

2000

GB

Peak

perfo

rman

ce /

node

: 147

.2 GF

LOPS

Main

memo

ry pe

r nod

e: 3

2 GB,

128 G

BDi

sk ca

pacit

y: 1

PB

AMD

Quad

Cor

e Opt

eron

2.3G

Hz

T2K-

Toda

i（

Hitac

hi HA

8000

-tc/R

S425

）To

tal P

eak p

erfor

manc

e: 5

4.9 T

FLOP

STo

tal nu

mber

of n

odes

: 56

Total

mem

ory

: 11

200 G

BPe

ak pe

rform

ance

/ no

de: 9

80.48

GFL

OPS

Main

memo

ry pe

r nod

e: 2

00 G

BDi

sk ca

pacit

y: 5

56 T

BIB

M PO

WER

7 3.8

3GHz

Yayo

i (H

itach

i SR1

6000

/M1)

Total

Pea

k per

forma

nce

: 1.13

PFL

OPS

Total

numb

er o

f nod

es: 4

800

Total

mem

ory

: 15

0 TB

Peak

perfo

rman

ce /

node

: 23

6.5 G

FLOP

SMa

in me

mory

per n

ode

: 32 G

BDi

sk ca

pacit

y: 1

.1 PB

+ 2.

1 PB

SPAR

C64 I

xfx1

.84GH

z

Oakle

af-fx

(Fuji

tsu P

RIME

HPC

FX10

)

(ret

ired,

Mar

ch 2

014)

“Oak

brid

ge-fx

” with

576

no

des

inst

alle

d in

Apr

il 20

14 (s

epar

ated

) (13

6TF)

•H

PC

I •

Sup

erco

mpu

ter S

yste

ms

in S

CD

/ITC

/UT

•O

verv

iew

of F

ujits

u FX

10 (O

akle

af-F

X)

•P

ost T

2K S

yste

m

3

Inno

vativ

e H

igh

Perf

orm

ance

C

ompu

ting

Infr

astr

uctu

re(H

PCI)

•H

PC

I–

Infra

stru

ctur

e (S

uper

com

pute

rs &

Dis

tribu

ted

Sha

red

Sto

rage

Sys

tem

)•

Sea

mle

ss a

cces

s to

K, S

C’s

(9 U

niv’

s), &

use

r's m

achi

nes

–P

rom

otio

n of

Com

puta

tiona

l Sci

ence

•S

trate

gic

Pro

gram

s fo

r Inn

ovat

ive

Res

earc

h (S

PIR

E)

–R

&D

for F

utur

e S

yste

ms

(Pos

t-pet

a/E

xasc

ale)

•H

PC

I Con

sorti

um–

Pro

vidi

ng p

ropo

sals

/sug

gest

ions

to th

e go

vern

men

t and

re

late

d or

gani

zatio

ns, o

pera

tions

of i

nfra

stru

ctur

e–

38 o

rgan

izat

ions

–O

pera

tions

sta

rted

in F

all 2

012

•ht

tps:

//ww

w.h

pci-o

ffice

.jp/

4

Nove

mbe

r 201

3

AICS

, RIK

EN：

K co

mpu

ter (

11.28

PF,

1.27P

iB)

Univ.

of T

okyo

：FX

10 (1

.13PF

, 150

TB)

SR16

000/M

1(54

.9TF,

10.94

TB)

T2K

(75.3

6TF,1

6TB/

140 T

F, 31

.25TB

)Ea

stHu

bPCC

lust

er(1

0TF,5

.71TB

/13TF

,8.15

TB)

GPU

Clus

ter(C

PU 4.

5TF,

GPU

16.48

TF,1.

5TB)

Wes

tHub

PCCl

uste

r(12.3

7TF,8

.25TB

)RE

NKEI

-VPE

:VM

Host

ing

Kyus

hu U

niv.：

FX10

(181

.6TF,

24TB

)CX

400 (

811.9

TF, 1

84.5T

B)HA

8000

-tc (7

12.5T

F, 24

.7TB)

SR16

000 V

M1 (8

.19TF

, 16T

B)

Hokk

aido

Univ.

：SR

1600

0/M1(

172T

F, 22

TB)

BS20

00 (4

4TF,

14TB

)

Nago

ya U

niv.：

FX10

(90.8

TF, 1

2TB)

CX40

0 (47

1TF,

43TB

)

Toho

ku U

niv.：

SX-9

(29.4

TF, 1

8TB)

Expr

ess5

800

(1.74

TF),

3TB)

Toky

o In

stitu

te o

f Tec

hnol

ogy

： TSUB

AME2

.5(5

.79 P

F, 15

0TB)

Osak

a Uni

v.：SX

-9 (1

6TF,

10TB

)SX

-8R

(5.3T

F, 3.3

TB)

PCCl

uster

(22.7

TF,

4.6TB

)

Kyot

o Un

iv.XE

6 (30

0.8 T

F, 59

TB)

Gree

nBlad

e800

0(24

2.5TF

, 38T

B)25

48X(

10.6T

F, 24

TB)

Univ.

of T

suku

ba：

T2K

(95.4

Tflo

ps, 2

0TB)

HA-P

ACS

(802

Tflo

ps,

34.3T

B)FI

RST

(36.1

TFlo

ps, 1

.6TB)

＞22

 PFLOPS

9 su

perc

ompu

ter c

ente

rs lo

cate

d at

un

iver

sitie

s in

Jap

anFi

scal

Year

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022

2023

Hokk

aido

Toho

ku

Tsuk

uba

Toky

o

Toky

oTe

ch.

Nago

ya

Kyot

o

Osak

a

Kyus

hu

The

next

nat

iona

l fla

gshi

p m

achi

ne

T2K

Toda

i(140

TF)

50+

PF

100+

PF

PostT

2K (2

0+ P

F, (1

00+

TiB, 6

00TiB

, 4.0

+ PB

/s, 0

.68+

PB/s)

)

Tsub

ame3

.0 (2

0+ P

F)Ts

ubam

e4.0

(100

+ PF

)

SX-8

+ S

X-9 (

21.7

TF, 3

.3 TB

, 50.4

TB/

s)(5

00+

TiB/s)

(5+

PiB/

s)

Cray

XC3

0 (4

00TF

)60

0TF

600T

F10

+ PF

50+

PF

(y

y)(

)(M

anyc

ores

ystem

) (70

0+ T

F)HA

-PAC

S (8

00 T

F)

Hitac

hi SR

1600

0/M1 (

172 T

F, 22

TB)

Clou

d Sys

tem H

itach

i BS2

000 (

44TF

, 14T

B)10

+ PF

50+

PF

()

Hitac

hi SR

1600

(25T

F)

Fujits

u FX1

0（27

0TF,

65.28

TB/

s), C

X400

(510

TF, 1

52.5

TiB, 1

51.14

TB/

s), G

PGPU

(256

TF, 3

0 TiB

, 53.9

2 TB/

s)

10+

PFHi

tachi

HA8

000tc

/HT2

10(5

00TF

, 215

TiB

, 98

.82TB

/s), X

eon P

hi (2

12TF

, 26.2

5 TiB

, 67

.2 TB

/s), S

R160

00(8

.2TF,

6 T

iB, 4

.4 TB

/s)50

+ PF

()

NEC

SX-9

+ E

xp58

00

(31T

F)~1

PFlop

/s30

+Pflo

p/s

Tsub

ame2

.5 (5

.5+ P

F,

110+

TB,

1160

TB/

s)Ts

ubam

e2.0

(2.4P

F,

97TB

, 744

TB/

s)

Cray

XE6

(30

0TF,

92.6T

B/s),

Gree

nBlad

e80

00(2

43TF

, 61.5

TB/

s)

50+

PFlo

psFu

jitsu F

X10

(90.8

TF, 3

1.8 T

B/s),

CX4

00(4

70.6T

F, 55

TB

/s)

Fujits

u FX1

0 (1

PFlop

s, 15

0TiB

, 408

TB/

s),

Hitac

hi SR

1600

0/M1

(54.9

TF,

10.9

TiB, 5

.376 T

B/s)

Fujits

u M90

00(3

.8TF,

1TB

/s)

HX60

0(25

.6TF,

6.6T

B/s)

FX1(

30.7T

F, 30

TB/

s)Up

grad

e (3.6

PF)

6c/

o Y.

Ishi

kaw

a (U

.Tok

yo)

SPIR

E/H

PCI

Stra

tegi

c Pr

ogra

ms

for I

nnov

ativ

e R

esea

rch

•O

bjec

tives

–S

cien

tific

resu

lts a

s so

on a

s K

com

pute

r sta

rts it

s op

erat

ion

–E

stab

lishm

ent o

f sev

eral

cor

e in

stitu

tes

for c

omp.

sci

ence

•

Ove

rvie

w

–S

elec

tion

of th

e fiv

e st

rate

gic

rese

arch

fiel

ds w

hich

will

cont

ribut

e to

find

ing

solu

tions

to s

cien

tific

and

soc

ial I

ssue

s•

Fiel

d 1:

Life

sci

ence

/Dru

g m

anuf

actu

re

•Fi

eld

2: N

ew m

ater

ial/e

nerg

y cr

eatio

n •

Fiel

d 3:

Glo

bal c

hang

e pr

edic

tion

for d

isas

ter p

reve

ntio

n/m

itiga

tion

•Fi

eld

4: Mono-zukuri

(Man

ufac

turin

g te

chno

logy

) •

Fiel

d 5:

The

orig

in o

f mat

ters

and

the

univ

erse

–

A n

atio

n w

ide

rese

arch

gro

up is

form

ed b

y ce

nter

ing

the

core

or

gani

zatio

n of

eac

h re

sear

ch a

rea

desi

gnat

ed b

y M

EX

T.

–Th

e gr

oups

are

to p

rom

ote

R&

D u

sing

K c

ompu

ter a

nd to

co

nstru

ct re

sear

ch s

truct

ures

for t

heir

own

area

7

HPC

I戦略

プログラム

Stra

tegi

c Pr

ogra

ms

for I

nnov

ativ

e R

esea

rch

8

•H

PC

I•

Supe

rcom

pute

r Sys

tem

s in

SC

D/IT

C/U

T•

Ove

rvie

w o

f Fuj

itsu

FX10

(Oak

leaf

-FX

)

•P

ost T

2K S

yste

m

9

Cur

rent

Sup

erco

mpu

ter S

yste

ms

Uni

vers

ity o

f Tok

yo•

Tota

l num

ber o

f use

rs ~

2,0

00•

Hita

chi H

A80

00 C

lust

er S

yste

m (T

2K/T

okyo

) (20

08.6

-20

14.3

)–

Clu

ster

bas

ed o

n A

MD

Qua

d-C

ore

Opt

eron

(Bar

celo

na)

–14

0.1

TFLO

PS

•H

itach

i SR

1600

0/M

1 (Y

ayoi

) (20

11.1

0-)

–P

ower

7 b

ased

SM

P w

ith 2

00 G

B/n

ode

–54

.9 T

FLO

PS

•Fu

jitsu

PR

IME

HP

C F

X10

(Oak

leaf

-FX

) (20

12.0

4-)

–S

PA

RC

64 IX

fx–

Com

mer

cial

ver

sion

of K

com

pute

r–

1.13

PFL

OP

S (1

.043

PFL

OP

S fo

r LIN

PA

CK

, 36t

hin

43r

dTO

P50

0)–

Add

ition

al 5

76 N

odes

with

136

TF

(Oak

brid

ge-F

X, 2

014.

04-)

10

11

His

tory

of W

ork

Rat

io

Res

earc

h A

rea

base

d on

CPU

Hou

rsFX

10 in

FY.

2013

(201

3.4~

2014

.3E)

12

Engine

ering

Earth/Space

Material

Energy/Physic

sInform

ation Sci.

Education

Indu

stry

Bio

Econ

omics

Serv

ice

Fee

•N

ot F

RE

E•

Ser

vice

Fee

= C

ost f

or E

lect

ricity

(Sys

tem

+A/C

)–

2M U

SD

for O

akle

af-F

X (2

MW

)–

1M U

SD

for T

2K (1

MW

) (~M

arch

201

4)

13

Serv

ices

for I

ndus

try

•O

rigin

ally

, onl

y ac

adem

ic u

sers

hav

e be

en a

llow

ed to

ac

cess

our

sup

erco

mpu

ter s

yste

ms.

•S

ince

FY

.200

8, w

e st

arte

d se

rvic

es fo

r ind

ustry

–m

ainl

y fo

r spr

ead

of la

rge-

scal

e pa

ralle

l com

putin

g–

not c

ompe

te w

ith p

rivat

e da

ta c

ente

rs, c

loud

ser

vice

s …

–ba

sica

lly, r

esul

ts m

ust b

e op

en to

pub

lic

–m

ax 1

0% to

tal c

omp.

reso

urce

is o

pen

for u

sage

by

indu

stry

–

spec

ial q

ualif

icat

ion

proc

esse

s ar

e ne

eded

•C

urre

ntly

onl

y O

akle

af-F

X is

ope

n fo

r ind

ustry

–N

orm

al u

sage

(mor

e ex

pens

ive

than

aca

dem

ic u

sers

)–

Tria

l usa

ge w

ith d

isco

unt r

ate

–R

esea

rch

colla

bora

tion

with

aca

dem

ic ra

te–

4 gr

oups

(FY

.201

4)–

Ope

n-S

ourc

e/In

-Hou

se C

odes

(NO

ISV

/Com

mer

cial

App

.)

14

Educ

atio

n•

Oak

leaf

-FX

onl

y•

2-D

ay “H

ands

-on”

Tut

oria

ls fo

r Par

alle

l Pro

gram

min

g by

Fac

ulty

Mem

bers

of S

CD

/ITC

(Fre

e)–

Fund

amen

tal M

PI (

3 tim

es p

er y

ear)

–A

dvan

ced

MP

I (2

times

per

yea

r)

–O

penM

Pfo

r Mul

ticor

eA

rchi

tect

ures

(2 ti

mes

per

yea

r)–

Par

ticip

ants

from

indu

stry

are

acc

epte

d.•

Gra

duat

e/U

nder

grad

uate

Cla

sses

with

Sup

erco

mpu

ter

Sys

tem

(Fre

e)–

We

enco

urag

e to

facu

lty m

embe

rs to

intro

duce

han

ds-o

n tu

toria

l of s

uper

com

pute

r sys

tem

into

gr

adua

te/u

nder

grad

uate

cla

sses

. –

Up

to 1

2 no

des

of O

akle

af-F

X–

Pro

posa

l–

Not

lim

ited

to C

lass

es o

f the

Uni

vers

ity o

f Tok

yo

15

•H

PC

I •

Sup

erco

mpu

ter S

yste

ms

in S

CD

/ITC

/UT

•O

verv

iew

of F

ujits

u FX

10 (O

akle

af-F

X)

•P

ost T

2K S

yste

m

16

Feat

ures

of F

X10

(Oak

leaf

-FX)

•W

ell-B

alan

ced

Sys

tem

–1.

13 P

FLO

PS

for P

eak

Per

form

ance

–M

ax. P

ower

Con

sum

ptio

n <

1.40

MW

•<

2.00

MW

incl

udin

g A

/C

•6-

Dim

. Mes

h/To

rus

Inte

rcon

nect

–H

ighl

y S

cala

ble

Tofu

Inte

rcon

nect

–5.

0x2

GB

/sec

/link

, 6 T

B/s

ec fo

r Bi-S

ectio

n B

andw

idth

•H

igh-

Per

form

ance

File

Sys

tem

–FE

FS (F

ujits

u E

xaby

te F

ile S

yste

m) b

ased

on

Lust

re•

Flex

ible

Sw

itchi

ng b

etw

een

Full/

Par

tial O

pera

tion

•K

com

patib

le !

•O

pen-

Sou

rce

Libr

arie

s/A

pplic

atio

ns•

Hig

hly

Sca

labl

e fo

r bot

h of

Fla

t MP

I and

Hyb

rid

17

FX10

Sys

tem

(Oak

leaf

-FX)

•A

ggre

gate

mem

ory

band

wid

th: 3

98 T

B/s

ec.

•Lo

cal f

ile s

yste

m fo

r sta

ging

with

1.1

PB

of c

apac

ity a

nd 1

31G

B/s

ec o

f ag

greg

ate

I/O p

erfo

rman

ce(fo

r sta

ging

)•

Sha

red

file

syst

em fo

r sto

ring

data

with

2.1

PB

and

136

GB

/sec

.•

Ext

erna

l file

sys

tem

: 3.6

PB

18

SPA

RC

64TM

IXfx

19

CPU

SPA

RC

64™

IXfx

1.84

8 G

Hz

SPA

RC

64™

VIIIf

x2.

000

GH

z

Num

ber o

f Cor

es/N

ode

168

Siz

e of

L2

Cac

he/N

ode

12 M

B6

MB

Pea

k P

erfo

rman

ce/N

ode

236.

5 G

FLO

PS

128.

0 G

FLO

PS

Mem

ory/

Nod

e32

GB

16 G

B

Mem

ory

Ban

dwid

th/N

ode

85 G

B/s

ec (D

DR

3-13

33)

64 G

B/s

ec (D

DR

3-10

00)

Rac

ks•

A “S

yste

m B

oard

” with

4 n

odes

•A

“Rac

k” w

ith 2

4 sy

stem

boa

rds

(= 9

6 no

des)

•Fu

ll S

yste

m w

ith 5

0 R

acks

, 4,8

00 n

odes

20

Tofu

Inte

rcon

nect

•N

ode

Gro

up–

12no

des

–A

/C-a

xis:

on

syst

em b

oard

, B-a

xis:

3sy

stem

boa

rds

•6D

：（X

,Y,Z

,A,B

,C）

–A

BC

3D

Mes

h: c

onne

cts

12 n

odes

of e

ach

node

gro

up–

XY

Z 3D

Mes

h：co

nnec

ts “A

BC

3D

Mes

h” g

roup

21

Softw

are

of F

X10

22

Com

putin

g/In

tera

ctiv

eN

odes

Logi

n N

odes

OS

Spe

cial

OS（X

TCO

S）R

ed H

at E

nter

pris

e Li

nux

Com

pile

r

Fujit

suFo

rtran

77/

90

C／

C++

GN

UG

CC，

g95

Fujit

su (C

ross

Com

pile

r)Fo

rtran

77/

90

C／

C++

GN

U(C

ross

Com

pile

r)G

CC，

g95

Libr

ary

Fujit

suS

SL

II (S

cien

tific

Sub

rout

ine

Libr

ary

II)，

C-S

SL

II，S

SL

II/M

PI

Ope

n S

ourc

eB

LAS，

LAPA

CK，

ScaL

APAC

K，FF

TW，

Supe

rLU，

PETS

c，M

ETIS，

Sup

erLU

_DIS

T，P

aral

lel N

etC

DF

App

licat

ions

Ope

nFO

AM，

AB

INIT

-MP，

PHAS

E，

Fron

tFlo

w/b

lue

Fron

tSTR

，R

EVO

CAP

File

Sys

tem

FEFS

(bas

ed o

n Lu

stre

)

Free

Sof

twar

eba

sh, t

csh,

zsh

, em

acs,

aut

ocon

f, au

tom

ake,

bzi

p2, c

vs, g

awk,

gm

ake,

gzi

p,

mak

e, le

ss, s

ed, t

ar, v

imet

c.

NO

ISV/

Com

mer

cial

App

licat

ions

(e.g

. NA

STR

AN

, AB

AQ

US,

AN

SYS

etc.

)

•H

PC

I•

Sup

erco

mpu

ter S

yste

ms

in S

CD

/ITC

/UT

•O

verv

iew

of F

ujits

u FX

10 (O

akle

af-F

X)

•Po

st T

2K S

yste

m

23

Post

T2K

Sys

tem

•20

-30

PFL

OP

S, F

Y.2

015

•M

any-

core

bas

ed (e

.g. (

only

) Int

el M

IC/X

eon

Phi

)•

Join

t Cen

ter f

or A

dvan

ced

Hig

h P

erfo

rman

ce

Com

putin

g (J

CA

HP

C, h

ttp://

jcah

pc.jp

/)–

Uni

vers

ity o

f Tsu

kuba

–U

nive

rsity

of T

okyo

24

•P

rogr

amm

ing

is s

till d

iffic

ult,

alth

ough

Inte

l com

pile

r wor

ks.

–(M

PI +

Ope

nMP

)–

Tuni

ng fo

r per

form

ance

(e.g

. pr

efet

chin

g) is

ess

entia

l–

Som

e fra

mew

ork

for h

elpi

ng u

sers

ne

eded