2015/5/31 Fifty Years of Japanese HPC ─From transistor to Exascale─ Yoshio Oyanagi Education...

23/04/21 1

Fifty Years of Japanese HPC─From transistor to Exascale─

Yoshio OyanagiEducation Center for Computational Sciences

Kobe University

Fifty years of Japanese HPC

Outline of my talk Historical overview of HPC in Japan, as

contrasted to US, Europe, China, ….. What is the difference between Japan and US

trends?– In Japan, big electronic companies manufactured

vector and parallel supercomputers besides main frames.

– In US, venture companies were active in vector (Cray, Convex etc.) as well as parallel (TMC, FPS, Meiko, nCUBE, KSR, ….) machines.

23/04/21 2Fifty years of Japanese HPC

“50 Years of HPC”A Column in HPCwire Japan

(Since July, 2014. Updated every Monday.)

23/04/21 3

http://www.hpcwire.jp(Sorry in Japanese)


1960’s in Japan

OKITAC-5090 (1961) Transistors + Diodes +

core memory

Floating point (decimal 10+2)– Add/sub 0.4 ms– Mult 4.9 ms (peak 0.4 kFlops)

23/04/21 4

Not very high performance !!


http://museum.ipsj.or.jp/computer/dawn/0037.html

1960’s in US CDC6600 IBM System/360

– 4 Mflops (peak)

Contract with DARPA for the ILLIAC IV

23/04/21 5

Pictures from Wikipedia


1960’sJapan USA

FACOM 270 HITAC 5020, 8000 TOSBAC 3400 NEAC 2200 MELCOM 3100, 9100 OUK 9400 FACOM 230-60

IBM 360 model 91 (1967)

IBM 360 model 85 (1968)

Intel founded (1968) IBM 2938 array proc.

(1969) CDC7600 (1969) AMD founded (1969)


23/04/21 7

Primordial Ages(1970’s)Japan USA/Europe

1970 IBM System/370

1971 CDC STAR-100 announced

1972 Goodyear STARAN

1974 DAP, BSP and HEP started

1975 ILLIAC IV operational

1976 Cray-1 shipped to LLNL

1976 FPS AP-120B

1977 Siemens SMS-201

1979 HEP single processor operational

(red: vector architecture)

1970 FACOM 230-75

1977 FACOM 230-75 APU

1978 HITAC M-180 IAP

1978 PAX project started

1979 HITAC M-200H IAP

1979 MELCOM COSMOIIIIAP

1981 MITI Supercomputer Project started( ～ 89)

(green: parallel architecture)Fifty years of Japanese HPC

23/04/21 8

Characteristics of Japanese vectors1. Japanese vector processors only a couple of years after Cray-1.

2. Manufactured by main-frame vendors with semiconductor facilities (not ventures)

Vector processors are attached to mainframes

3. FACOM 230-75 APU (first Japanese vector)

a) CPU and APU share the main memory (1 MW×36bits)Heterogeneous architecture

b) Vector register of 1792 words

c) list vector and DO-loop with IF supported

d) 22 Mflops (single), 11 Mflops (double)

e) installed in NAL in August 1977


Characteristics of Japanese vectors4. HITAC IAP (1978-)

a) memory-to-memory (no vector registers)

b) summation, inner product and 1st order recurrence could be vectorized in FORTRAN

c) vectorization of loops with IF’s (M280)

d) M-200H IAP (1979): 48 Mflops,

M-280H IAP (1982): 67 MFlops

5. NEC ACOS 1000 IAP (1982)a) 36 bit machine

b) peak 28 MFlops


23/04/21 10

1st Generation (1H of 1980’s)Japan US/others

1981 CDC Cyber 2051982 Cosmic Cube Project1982 Cray X-MP/2 (420 MF)1982 Alliant FX/8 delivered1982 HEP installed1983 Encore, Sequent and TMC

founded, 1983 ETA span off from CDC1984 Cray X-MP/4 (820 MF)1984 Legend group in China started1985 Convex C11985 Intel iPSC/1, T414, nCUBE/1,

Stellar, Ardent1985 Cray-2 (1.952 GF)1986 CM-1 shipped1986 FPS T-series

1980 PAX-32

1981 MITI Supercomputer

Project ( ～ 89)

1982 Fifth Generation Project ( ～ 92)

1982 NEC ACOS-1000 IAP

1982 HITAC M280H IAP

1983 HITAC S-810/20 (630 MF)

1983 FACOM VP-2001984 PAX-64J

1985 FACOM VP-400 (1.142 GF)

1985 NEC SX-2 (1.3 GF)


23/04/21 11

Characteristics of Japanese SC in the 1st Generation

1. Compatibility with the mainframes

2. Single processor with multiple pipes

3. Large main memory (256MB vs. X-MP 32MB)

4. Large vector registers

5. List-directed vector instructions (gather/scat)

6. Different control of vector units

7. No commercial parallel machines


Characteristics of Japanese SC’s in the 1st Generation

8. Aggressive installation of SC’s in universities and laboratories

– 7 SC centers open to all university people– Laboratory SC’s for physics, chemistry, …

In US, most SC’s are in DOE, DOD, NASA,not available for university researchersLax Report (1982)

– 5 NSF centers in 1985-6


23/04/21 13

Characteristics of Japanese SC’s in the 1st Generation

9. MITI Supercomputer Project (1981-89fy) Started before the Japanese supers (S810, VP,

SX) Targets:

– 10 GFlops parallel vector machine (PHI),– Dataflow machine (sigma-1)– Josephson device, MPP etc.

In alliance with six companies


23/04/21 14

2nd Generation (2H of 1980’s)Japan US and Europe etc.

1986 “863” Plan in China1986 Manheim Supercomputer

seminar (→ISC)1987 ETA-10 (10 GF)1987 CM-21988 Cray Y-MP (2.66 GF)1988 Intel ipsc/21988 First Supercomputing

Conference in Orlando1989 ETA shut down, JvN SC

shut down1989 BBN TC2000, Myrias

SPS-2, Meiko CS, NCube21990 Intel ipsc/860, MasPar

MP-1

1987 HITAC S-820 (3 GF)1989 FACOM VP2600 (5 GF)1990 MITI Supercomputer

Project ended1990 NEC SX-3 (22 GF)1990 QCDPAX completed


23/04/21 15

Characteristics of Japanese SC in the 2nd Generation

1. Vector multiprocessor appeared in Japan

with modest multiplicity 2-4 vs. 8 (Y-MP, ETA)

2. The semiconductor technology developed for the vector is transferred to mainframes

(Used to be: mainframe→vector)

3. Still no commercial MPP’s in Japan, although parallel research was active in academia


Installation of SC’s in Japan (1987-8) Cray Research (Cray-1, X-MP):10 ETA (ETA-10): 1 Hitachi (S810):15 NEC (SX-1/2):7 Fujitsu (VP-50, 100, 200, 400):36


US-Japan Trade Conflicts

1985/9: Plaza Accord (G5) 1985: SX-2 was cancelled by NCAR after

bidding 1986/9: US-Japan Semiconductor Agreement 1987: Japanese SC was cancelled by MIT after

bidding


23/04/21 18

Super 301 Act

In 1989, US government decided to apply “Super 301 Act” (Omnibus Trade and Competitiveness Act of 1988) to Japan and identified three items including supercomputers.

Washington threatened Japanese governmental institutions to purchase US supercomputers. (Titech ETA10, ETL X-MP)

On the other hand, US governmental institutions had no Japanese supercomputers at all.


23/04/21 19

3rd Generation (1H of 1990’s)Japan USA/Europe

1990 MasPar MP-11990 Intel iPSC/8601991 HPCC started (-96)1991 Cray Y-MP C90 (16 GF)1991 Intel Paragon, TMC CM-51992 FPS bankrupt 1992 MasPar MP-21993 Top500 started1993 HPCN started ( ～ 2001)1993 Cray T3D, CS6400, nCUBE2, KSR11993 SSI shut down1993 IBM SP-11993 Cray-31994 SP-21994 TMC, KSR Chapter 11

1992 RWCP started

1992 CP-PACS started

1993 Fujitsu NWT

1993 Fujitsu VPP500

1993 HITAC S-3800 (32 GF)

1993 NEC Cenju-3

1994 Fujitsu AP1000

1995 NEC SX-4


23/04/21 20

Characteristics of Japanese SC in the 3rd Generation

Drastic changes in Japanese vectors1. Hitachi (S3800)

Shared memory vector parallel processorusing ECL technology

2. Fujitsu (VPP500)Distributed memory vector parallel proc.using GaAs as well as silicon tech.

3. NEC (SX-3)Cluster of shared memory vector parallelUnix as host OS

4. No action to “Attack of killer micros.”(SC90)

◆ Commercial MPPs, sold as a parallel testbed


Trends in US (1st half of 1990’s) Active vector machines: Cray Y-MP, C90 and

Convex C2 MPP:

Performance of COTS CPU’s was increasing drastically custom CPU → commodity CPU chips ventures company → big companies (IBM, Cray, Intel)

Strong national initiative:HPCC, NII, HPC Act 1991


HPCC in US Blue Book (1991/2, G.W.Bush)

– “Grand Challenges: High Performance Computing and Communications”

The High Performance Computing Act of 1991 Trade conflicts:

– “Japan should buy more US supers” (M. Kantor, 1993/4) Otherwise, possible retaliation.

– 6 out of 11 in public sector bought US machines in the 1994 supplementary budget.

In Europe, Rubbia Report (1991/1)– European Teraflop Initiative


23/04/21 23

4th Generation (2H of 1990’s)Japan USA/Europe/China

1995 DOE: ASCI started1995 Cray T90 (57.6 GF)

1995 Cray Computer bankrupt1996 Dawning in China started1996 Cray T3E 1996 Cray Research merged into SGI1996/10 Serymour died1996 nCUBE merged into Oracle1996 MasPar went out of HPC1997 ASCI Red (Intel)1997 NSF: NPACI started1998 Cray SV11998 ASCI Blue Pacific (IBM)1998 ASCI Blue Mountain (SGI)

1995 Fujitsu VPP300

1996 cp-pacs completed

1996 Hitachi SR2201

1996 Fujitsu VPP700

1996 Fujitsu AP30001997 NEC Cenju-41998 Hitachi SR80001998 NEC SX-5


23/04/21 24

Characteristics of Japanese SC in the 4th G.

1. Fujitsu and NEC seem to follow the line of conventional vector supercomputers in CMOS

Fujitsu: distributed memory

NEC: cluster of shared memory nodes

2. Hitachi adopted RISC (pseudo)vector architecture.

SR2201: sliding window (proposed by Tsukuba)

SR8000: 8 tightly coupled CPUs in one node

(self-developed CPU with Power architecture)


US Trends ASCI Project (originally 1995-2004)

– SC’s in LANL, LLNL, SNL– Red (1+), Blue (3+), White (10+), Q(30+),

Purple (100+) --- every 2 years PITAC (I:1997-2001, II: 2003-2005)

– IT2 project NPACI (1997-2004) Petaflops I in Pasadena (1994)

Petaflops II in Santa Barbara (1999)


NCAR Procurement 1996/5/20 UCAR selected SX-4 for NCAR

– finalists were C90, SX-4, VPP700 Criticism by Cray and Obey

– Tax from US people should be used for US competitiveness

– NEC (and Fujitsu) was dumping 1996/7/29 Cray filed an apeal to DoC and ITC for

dumping 1997

– 454% anti-dumping customs to NEC supercomputers

– 388.7% to Fujitsu


23/04/21 27

9306 9311 9406 9411 9506 9511 9606 9611 9706 9711 9806 9811 9906 9911 0006 0011 0106 01111 NWT NWT NWT NWT Todai cp- p2 NWT NWT NWT cp- p3 Todai NWT4 Todai cp- p Todai5 NEC ATP ATP KEK KEK Todai LRZ Todai6 AES NEC Tsuku Tsuk Todai cp- p KEK7 AES Riken Riken J AERI KEK NWT LRZ Todai8 KEK ECMW Osak9 NEC Kyush Todai KEK

10 HitachHitac NEC ECMW11 Todai Todai J AERI Stutt12 NEC Nagoy AES TAC ECMWLRZ Osak13 Toho Toho Todai14 ATP Gene J AERI Todai cp- p KEK15 AES Tsuk ISS Nagoy NWT Kyoto16 NEC Riken ECMW17 Toho Toho Gene NEC Kyush TAC ISS Todai LRZ18 AES Toho ISS Osaka KEK AES cp- p J MA19 IMS AES ATP Osaka ECMWFZJ KEK20 IMS Tsuk Stutt


Japanese machines in Top 20

23/04/21 28

Fifth Generation (1H of 2000)Japan US/Europe

2000: Tera became Cray

2000: SGI: Origin 3000

2000: ASCI White (LLNL, IBM) 12 TF

2001: NSF: TeraGrid stared

2002: ASCI Q (LANL, HP) 20TF

2002: DOD: HPCS started

2002: Cray (NEC) SX-6 to ARSC

2004: NASA Columbia (SGI) 64TF

2004: BlueGene/L at IBM 90TF

2004: “The Path to Extreme Computing” conf. (to Exa)

2005: BlueGene/L (360TF)

2005: ASCI Red Storm (SNL)Cray XT3/XT4

2001: NEC: anti-dumping customs cleared

2001: Fujitsu PRIMEPOWER 2000

2001: NEC SX-6

2002: Earth Simulator 40TF

2002: Fujitsu PRIMEPOWER HPC2500

2003: NEC SX-7

2004: NEC SX-8

2004: Hitachi SR11000


Japan Trends NEC kept making vector machines after the Earth

Simulator Fujitsu: PRIMEPOWER (Sparc) and PRIMERGFY

(x86) Hitachi: OEM of IBM servers. SR11000 (Power4+,

6.8 GF/CPU) NAREGI (2003-8): Grid project


Japan Trends

IT Strategic Headquarter (2001): e-Japan, but only network was emphasized.– At this stage, level up of supercomputers was to be

promoted according to the needs of each field (not a national project).

Earth Simulator attained 36 Tflops (2002) Information Science and Technology committee in

Mext has been discussing the measures to promote computational science and technology since August 2004.

Fifty years of Japanese HPC 3023/04/21

Japanese machines in top 20

23/04/21 Fifty years of Japanese HPC 31

0206 0211 0306 0311 0406 0411 0506 0511 0606 0611 0706 0711 0806 0811 0906 0911 1006 1011 1106 1111 1206 1211 1306 13111 ES ES ES ES ES K K2 K3 ES K4 ES TIT K K5 TIT TIT6 NAL ES7 Riken TIT8 AIST9 TIT

10 ES11 J AXA TIT12 AIST Roku13 Todai TIT14 LRZ Riken ES TIT15 J AERI AIST TIT Todai Roku1617 KEK TIT18 KEK Todai19 Osak AIST20 ES Tsuk Roku

US Trends DOE: ASCI continues

– BlueGene and Red Storm added NSF: Teragrid (2001-10) DOD: HPCS (High Productivity Computing Systems)

– 2002: Phase I (IBM, Cray, Sun, HP, SGI)

– 2003: Phase II (Cray, IBM, Sun)

– 2006: Phase III (Cray: XC, IBM: PERCS) HEC Revitalization Act of 2004 and 2005 Exa started in “The Path to Extreme Computing” in

Santa Fe (2004/10)


Sixth Generation (2H of 2000 and later)Japan US/Europe/China

2006-12: K Computer P.– 2011: 10 PF attained

2006: T2K open SC– 2008: installed in 3 univ.

2006: TSUBAME 1.0– 2010: TSUBAME 2.0

2009: Fujitsu fx-1 to JAXA

Cray XT/XE/XC 2005: ASCI Purple (LLNL, IBM) 2006: Rangers (TACC) 2007: NCSA, BlueWaters started

(IBM→2011 Cray) 2007: Kraken (Tennessee) 2008: Roadrunner (LANL) 2010: Tianhe-1A (NUDT) 2010: PRACE started 2011: NSF, SXEDE 2013 America COMPETES Act


History of the K Computer Recommendation to a Mext committee (2005):

– Promote a national project to construct a leading edge supercomputer

– Government decision (July 25, 2005) Riken started the project (October 2005) Mext funded four projects to promote “Element

Technologies for Future Supercomputers” in 2005-2007. $40M per year (in total)– Four groups were accepted1. System Interconnect (Kyushu U and Fujitsu)2. Interconnect by IP (U of Tokyo, Keio U etc)3. Low Power Device and Circuits (Hitachi, U of Tokyo, U of Tsukuba)4. Optical Connection of CPU and Memory (NEC and Titech)


History of the K Computer 2005: Proposed architectures and killer applications 2007: Hybrid machine with vector and scalar 2009: Five strategic application fields defined 2009: Withdrawal of vector machine 2009: the Government Revitalization Unit proposed

to shutdown the Next Generation Supercomputer Project (Nov. 13, 2009)

2009: HPCI started(High Performance Computing Infrastructure, alliance of supercomputer centers and users)


Original ProposalLarge scale processing Scalar computer

Special-purpose computerFifty years of Japanese HPC 3623/04/21


Friday, November 13, 2009

The 3rd WG of the Government Revitalization Unit

“Why should it be No.1 in the world?”

“Is No.2 not enough?”

Vote ：Abolish １Postpone ６Budget shrink ５

ConclusionFreeze the project!

← 村田 ( 謝 ) 蓮舫 (Hsieh Lien Fang)Picture from www.rehabilitate.jp

3723/04/21

No.1 in the world 8.162 Pflops attained using 80% of the system

ー No.1 in the Top500 (ISC2011) June 20, 2011

10.51 Pflops in SC2011 (Seattle) Open to users: September 2012

– 1st period: 2012/9 to 2014/3– 2nd period: 2014/4 to 2015/3

RIST is to manage K Computer users(Research Organization for Information Science & Technology)


What’s next?Toward EXA FLOPS!!

Preliminary consideration among scientists(In US, since 2004)

Mext started a WG for future HPC (April 2011)– Hardware-System Software-Application co-design is

important.

– Should be science-driven.• We identified possible break throughts in science and technology.

Social needs are also considered.

• Linpack Exaflops is not our target.

– Limitation by budget, power, ….

– Several different architectures are considered.Fifty years of Japanese HPC 3923/04/21

Architecture

Algorithm

Application


Two subgroups worked

Application Subgroup– Application

– Numerical library

– Algorithm

– Automatic tuning

System Subgroup– CPU and architecture

– Compiler

– System software

Fifty years of Japanese HPC 41

Final Report in March 2012

23/04/21

Memory and memory bandwidthBig technical issue

Fifty years of Japanese HPC 42

Large B/FlopsSmall B/Flops

Large B/Flop

Small B/Flop

0.1

0.1standard

SoC

K

GPU

Small system

ES

23/04/21

WG for Future HPCI Policy

2012/4 ～ 2014/3 (25 meetings) 26 members, Chair: Oyanagi Recommendations

– Hierarchical deployment of supercomputers– Leading machines: Flagship system and its

complementary systems– Flagship: “exascale” machine toward 2020– Support to users, especially industry users


Flagship system

Complementary systems to the flagship

Universities and laboratories

Leading system

s

Allocated thru H

PCI


2014年 2015年 2016年 2017年 2018年 2019年 2020年

日本のスケジュール案

ハード・システム

アプリケーション

アメリカのスケジュール案

ハード

ソフト

ヨーロッパのスケジュール案（詳細不明）

システム

中国のスケジュール案

システム

　　基本設計　　　　　　　試作・詳細設計　　　　　製造（量産）　設置・調整　　　　運用

エクサ向けアプリケーション開発・利用

システム・ノード設計　　　　実験機　　　　　　　　　　　システム製造・設置

Ｐ環境設計　 P 環境構築　　　　　　↓アプリ開発開始　　　　　　　　　　　　↓ P 環境更新　　　

8th Framework Program (FP8) Horizon 2020

第 12 次 5 カ年計画　　　　　↓ 100 PF 　　　　　　　　　　第 13 次 5 カ年計画 1 EF ↓

Development plans of exascale machines in the world(preliminary)


9 Key topics for the post-K(Komiyama Committee)

Healthy longevity1. Innovative drug design

2. Integrated computational bioscience Disaster mitigation and environment

3. Earthquake and tsunami

4. Advanced prediction of climate and environment


9 Key topics for the post-K Energy problem

5. New technology for energy production, storage and usage

6. Innovative clean energy system Industry competitiveness

7. New functional devices and materials

8. Innovative design and manufacturing Basic science

9. Fundamental law and evolution of the universe


23/04/21 48

Observation of Japanese HPC (1/3) Japan has manufactured vector supercomputers since

1977 Japanese vector computers were designed as an

extension of main frames.– In the early stage, development cost was amortized among

mainframes (US-Japan conflicts). Stress of easiness of use (compilers and tools)

– Very good vectorizing compilers.– Users exploited the power of vectorization.– Users were spoiled by them.– Application software development not enough.


Observation of Japanese HPC (2/3) Until late 1990’s, Japanese vendors focused on vector

machines. Late entry to parallel architectures. Vendors thought parallel machines were for

specialized purposes (eg. image processing). Most users dared not try to harness parallel machines

in the 80’s.– Users found difficulties in using message passing.

Some computer scientists were interested in building parallel machines, but they were not used for practical scientific computing.


Observation of Japanese HPC (3/3) Practical parallel processing for scientific

computing was started by application users: QCDPAX, NWT, CP-PACS, GRAPE’s, ES.

After the K Computer, parallel users increased not only in academia but also in industry

We must prepare for the post-K exascale.– Hierarchical deployment– Home development– Co-design, Science-driven


2015/5/31 Fifty Years of Japanese HPC ─From transistor to Exascale─ Yoshio Oyanagi Education...

Documents

Transcript of 2015/5/31 Fifty Years of Japanese HPC ─From transistor to Exascale─ Yoshio Oyanagi Education...