Summary of “Towards Petaflops” Workshop, 24—28 May 1999 A D Kennedy, D C Heggie, S P Booth The...

Summary of “Towards Petaflops” Workshop, 24—28 May 1999

A D Kennedy, D C Heggie, S P BoothThe University of Edinburgh

Towards PetaflopsSummary


ProgrammeMonday24th May

Tuesday25th May

Wednesday26th May

Thursday27th May

Friday28th May

ComputationalChemistry

Particle Physics &Astronomy

Biological sciences Materials, soft & hardMeteorology &

Fluids09:00

09.30 REGISTRATIONDr N H Christ

Columbia University, USADr J M Levesque

IBM Research, USADr G Ackland

University of EdinburghDr P BurtonUKMO, Bracknell

10:00 WelcomeProf A D Kennedy

10:15TEA/COFFEE

10:30

Prof R CatlowRoyal Institution, London

11:15 TEA/COFFEEProf R D’Inverno

University of SouthamptonDr M F GuestCLRC, Daresbury

Dr M PayneCambridge University

Dr T ArberUniversity of St Andrews

11:30 Prof E De SchutterUniversity of Antwerp, Belgium

Dr R TripiccioneINFN-Pisa, Italy

Dr N TophamUniversity of Edinburgh

Dr P CoveneyQueen Mary & Westfield

College, London

Dr M RuffertUniversity of Edinburgh

12.30 LUNCH13.30 Discussion Session

Chair: Prof R CatlowDr A H NelsonUniversity of Cardiff

Prof M van HeelImperial College, London

Dr D RowethQSW, Bristol

Discussion SessionChair: Dr T Arber

14.15 Dr L G PedersenUniversity of North Carolina,

USA

Prof D C HeggieUniversity of Edinburgh

Mr M WoodacreSGI/Cray

Dr M F O’BoyleUniversity of Edinburgh

15.00 TEA/COFFEE15.30 Dr M Wilson

University of DurhamDiscussion SessionChair: Prof N Christ

Discussion SessionChair: Dr J Bard

Discussion SessionChair: Dr M Payne

16.15 Dr A R JenkinsUniversity of Durham

Dr C M ReevesUniversity of Edinburgh

17:00 CLOSE


Participants

Dr Graham Ackland graeme@holyroodProf Tony Arber [email protected] Jonathan Bard [email protected] Stephen Booth [email protected] Ken Bowler [email protected] Paul Burton pmburton.meto.gov.ukProf Mike Cates [email protected] Richard Catlow [email protected] Norman Christ [email protected] Peter Coveney p.v.coveney.qmw.ac.ukProf Ray D d' Inverno [email protected] Erik De Schutter [email protected] Paul Durham [email protected] Dietland Gerloff [email protected] Simon Glover [email protected] Bruce Graham [email protected] Martyn F Guest [email protected] Douglas C Heggie [email protected] Suhail A Islam [email protected] Adrian R Jenkins [email protected] Bruce Jones [email protected] Balint Joo [email protected] Anthony D Kennedy [email protected] Richard Kenway [email protected] Crispin Kneble [email protected]

Mr John M Levesque [email protected] Nick Maclaren [email protected] Rod McAllister [email protected] Avery Meiksin [email protected] Alistair Nelson [email protected] Mike O'Boyle [email protected] John Parkinson [email protected] Mike C Payne [email protected] Lee G Pederson [email protected] Nilesh Raj [email protected] Federico Rapuano Federico.Rapuano@roma1Dr Clive M Reeves [email protected] Duncan Roweth [email protected] Max Ruffert [email protected] Vance Shaffer [email protected] Doug Smith [email protected] Philip Snowdon [email protected] Nigel Topham [email protected] Arthur Trew [email protected] Raffaele Tripiccione [email protected] Marin Van Heel [email protected] Claudio Verdozzi [email protected] Mark Wilson mark.wilson.durham.ac.ukMr Stuart Wilson [email protected] Michael Woodacre [email protected] Andrea Zavanella [email protected]


Introduction

Objectives of this Summary Summarise areas of general agreement Highlight areas of uncertainty or disagreement Concentrate on technology, architecture, & organisation For details of science which might be done see the slides of

the individual talks This summary expresses views & opinions of its authors

– It does not necessarily represent a consensus or majority view...

– … but it tries to do so as far as possible


Devices & Hardware

Silicon CMOS will continue to dominate GaAs is still tomorrow’s technology (and always will be?)

Moore’s Law Performance increases exponentially Doubling time of 18 months Will continue for at least 5 years, and probably more Trade-offs between density & speed

– Gb DRAM and GHz CPU in O(5 years) ...– … but not both on the same chip

Choice between speed & power 10 transistors per device by 2005, 10 by 2012 Most cost-effective technology is usually a generation behind

the latest technology


Devices & Hardware

Memory latency will increase More levels of cache hierarchy

– Implies a tree-like hierarchy of access speeds

– Not clear how scientific HPC applications map onto this

Access to memory becoming relatively as slow as access to remote processor’s cache

Understanding of memory architecture required to achieve optimal performance

– analogous to use of virtual memory

In the fairly near future Arithmetic will be almost free Pay for memory & communications bandwidth


Devices & Hardware

Devices & HardwareTechnology driven by mass market Commodity parts “Intercepting technology”

– systems designed to use technology current under development

– cost & risk of newest generation v. performance benefit

– “sweet point” on technology curve

PCs Workstations DSP … not designed for HPC


Devices & Hardware

Level of integration HPC vendors will move from board to chip level design Cost effective to produce O(10³—10) chips Silicon compilers Time scale?


Devices & Hardware

Error rates will increase Fault tolerance required Implications for very large systems? Time scale?

Disks & I/O Increasing density Decreasing cost/bit Increasing relative latency


Architecture

Memory addressing Flat memory (implicit communications)

– Model that naïve users want

– Does not really exist in hardware

– Dynamic hardware coherency mechanisms seem unlikely to work well enough in practice

Distributed memory (explicit communications)– NUMA

– Protocols

• MPI, OpenMP,…

• SHMEM

– Scientific problems usual have a simple static communications structure, easily handled by get and put primitives


Architecture

Single node performance Fat nodes or thin nodes?

– Limited by communication network bandwidth?

– Limited by memory bandwidth (off-chip access)?

– “Sweet point” on technology curve

Single node architecture VLIW Vectors Superscalar Multiple CPUs on a chip


Architecture

Communications What network topology?

– 2d, 3d, or 4d grid network, butterfly, hypercube, fat tree

– Crossbar switch

Bandwidth Latency

– Major problem for coarse-grain machines

Packet size– A problem for very fine-grain machines


Architecture

MPP Scalable for the right kind of problems

– up to technological limits

Commercial interconnects– e.g., from QSW (http://www.quadrics.com/)

Flexibility v. price/performance– Custom networks for a few well-understood problems which

require high-end performance (e.g., QCD)

– More general networks for large but more general purpose machines


Architecture

SMP clusters Limited scalability? Appears to be what vendors want to sell to us

– IBM, SGI, Compaq,…

– Large market for general purpose SMP machines

– Adding cluster interconnect is cheap

Unclear whether large-scale scientific problems map onto the tree-like effective network topology well

How do we program such machines?


Architecture

PC or workstation clusters Beowulf, Avalon, ... Cheap, but not tested for large machines Communication mechanisms unspecified “Farms” of PCs very cost-effective solution to provide large

capacity

Static v. dynamic scheduling Static (compiler) instruction scheduling more appropriate than

dynamic (hardware) scheduling for most large scientific applications


Languages & Tools

Efficiency new languages will not be widely used for HPC unless they

can achieve performance comparable with low level languages (assembler, C, Fortran)

Portability to different parallel architectures to next generation of machines to different vendor’s architecture

Reusability Object-oriented programming Current languages not designed for HPC (C++, JAVA, …)


Languages & Tools

Optimisation Compilers can handle local optimisation well

– register allocation

– instruction scheduling

Global optimisation will not be automatic– choice of algorithms

– data layout

– memory hierarchy management

– re-computation v. memory use

– could be helped by better languages & tools


Languages & Tools

How to get scientists & engineers to use new languages & tools? Performance must be good enough Compilers & tools must be widely available Compilers & tools must be reliable Documentation & training

New generation of scientists with interest and expertise in both Computer Science and Applications required Encouragement to work & publish in this area Usual problems for interdisciplinary work

– credit for software written is not on par with publications


Models & Algorithms

Disciplines with simple well-established methods Models are “exact” Methods are well understood & stable Errors are under control

– at least as well as for experiments

Leading-edge computation required for international competitiveness

Examples:– Particle physics (QCD)

– Astronomy (N body)


Models & Algorithms

Disciplines with complex models Approximate models used for small-scale physics Is reliability limited by

– sophistication of underlying models?

– scale of computation?

– availability of data for initial or boundary conditions?

Many different calculations for different systems– capacity v. capability issues

Examples:– Meteorology

– Materials


Models & Algorithms

Reliance on packages Commercial packages not well-tuned for large parallel

machines– Algorithms may need changing

Community has resistance to changing to new packages or writing their own systems

Examples:– Chemistry

– Engineering


Models & Algorithms

Exploration HPC not widely used Access to machine and expertise is a big hurdle Exciting prospects for future progress Algorithms and models need development Examples:

– Biology


Access & Organisation

Bespoke machines The best solution for a few special areas QCDSP, APE

Special-purpose machines Grape



Performance versus cost Slide courtesy of Norman

Christ (Columbia University) Diagonal lines are fixed cost Note dates of various

machines



Commercial machines SMP

– Convenient, easy to use, but not very powerful

– Good for capacity as opposed to capability

SMP clusters– Unclear how effective for large-scale problems

– Unclear how they will be programmed

Commercial interconnects (QSW,…)



Capacity v. capability Large machine required to get “final point on graph” Cost-effectiveness International competitiveness



Shared v. dedicated resources Systems management costs

– advantages of shared resources

• central large-scale data store

• backups

– disadvantages

• more reasonable requests if users have to pay for their implementation

• tendency for centres to invent software projects which are not the users’ highest priority



Dedicated machines for consortia Flexibility in scheduling

– Do not have to prioritise projects in totally different subjects

– Users know & can negotiate with each other

Ease of access for experimental projects– consortia can be more flexible at allocating resources for

promising new approaches


Sponsors

The workshop was supported by The University of Edinburgh Faculty of Science & Engineering Hitachi IBM


Glossary

API Application Program Interface

– documented interface to a software subsystem so that its facilities can be used by application (user) programs

Butterfly Network Network topology which allows “perfect shuffle” required for

FFTs to be carried out in parallel– equivalent to network, Fat tree, and Hypercube


Glossary

Cache Fast near-processor memory (usually SRAM) which contains a

copy of the contents of parts of main memory– often separate instruction & data caches

– commonly organised as a hierarchy of larger caches of increasing latency

– data is automatically fetched from memory when needed if it is not already in the cache

– an entire cache “line” is moved from/to memory even if only part of it is required/modified

– data is written back to memory when cache “line” is needed for data from some other memory address, or when someone else needs the new value of the data

– one (direct map) or several (set associative) cache “lines” can be associated with a given memory address


Glossary

Capability The ability to solve on big problem in a given time

Capacity The ability to solve many small problems in a given time

CISC Complex Instruction Set Computer

– instructions combine memory and arithmetic operations

– instructions are not of uniform size or duration

– often implemented using microcode

– power is dissipated only when switching states


Glossary

CMOS Complementary Metal Oxide Semiconductor

– by far the most widely used VLSI technology at present– VLSI technology in which both p type and n type FETs are used– design methodology is that each output is connected by a low

impedance path to either the source or drain voltage, thus low static power dissipation

– NMOS technology requires less fabrication steps, but draws more current and is not in common use anymore

– BiCMOS allows the construction of both FET and Bipolar transistors on the same chip

• requires more fabrication steps and therefore larger minimum feature size (and thus lower density) for an acceptable yield

• bipolar transistors can drive more current (for a given size and delay) than FETs


Glossary

Coherency Means of ensuring that the copies of data in memory and

caches are consistent– the illusion that a single value is associated with each address

Crossbar Network topology allowing an arbitrary permutation in a single

operation

Data Parallel Programming model in which all nodes carry out the same

operation on different data simultaneously– may be implemented using SIMD or MIMD architectures


Glossary

Delayed branches several instructions following a branch are unconditionally

executed before the branch is taken, allowing the instruction pipeline to remain filled

DRAM Dynamic RAM

– each bit is stored as the charge on the gate of an FET transistor

– only one transistor required to store each bit

– DRAM needs to be refreshed (read and rewritten) every few s before charge leaks away


Glossary

DSP Digital Signal Processor

– low cost

– low power

– no cache

– used for embedded devices

Dynamic Scheduling Order in which instructions are issued is determined on the

basis of current activity– dynamic branch prediction: instructions are prefetched along the

path taken the last few times through this branch

– scoreboarding: instructions are delayed until the resources required (e.g., registers) are free


Glossary

ECC Error Correcting Codes

– mechanism for correcting bit errors in DRAM by using, e.g., Hamming codes

Fat Nodes Fast & large processor nodes in a multiprocessor machine

– allow relatively large sub-problem to live on each node– permits coarse-grained communications (large packets, but less

of them)– memory bandwidth is a potential problem

Fat Tree Network topology which allows “perfect shuffle” required for

FFTs to be carried out in parallel– equivalent to Butterfly, network, and Hypercube


Glossary

FET Field Effect Transistor

– transistor in which the channel (source to drain) impedance is controlled by the charge applied to the gate

– no current flows from the gate to the channel

– c.f., a bipolar transistor in which a current is drawn flows through the base from the emitter to the collector

FFT Fast Fourier Transform

– O(n log n) algorithm for taking the Fourier transform of n data values

Glossary


Glossary

GaAs Gallium Arsenide

– a semiconductor whose band gap structure allows faster switching times than those for Si (silicon)

– fabrication technology lags that for Si

– larger minimum feature size for acceptable yield

– VLSI speed is limited by path length, not by switching times

Generation a level of technology used for chip fabrication, usually

measured by the minimum feature size

GRAPE special purpose machine for solving the n-body problem


Glossary

Hypercube Network topology which allows “perfect shuffle” required for

FFTs to be carried out in parallel– nodes live on the vertices of a d-dimensional hypercube

– communications links are the edges of the hypercube

– equivalent to Butterfly, network, and Fat tree

Instruction prefetch The ability to fetch & decode an instruction while the previous

instruction is still executing– allows “pipelining” of instruction execution stages

– requires special techniques to deal with conditional branches where it is not yet known which instruction is “next”


Glossary

Latency The time between issuing a request for data and receiving it

Microcode sequence of microinstructions used to implement a single

machine instruction– similar functionality to having simpler (RISC) instructions with an

instruction cache, but less flexible

– stored in ROM

– reduces complexity of processor logic

– some similarities

• “vertical” code RISC

• “horizontal” code VLIW


Glossary

MPI Message Passing Interface

MPP Massively Parallel Processor

– a collection of processor nodes each with their own local memory– explicit distributed memory architecture– nodes connected by a network of some regular topology

MIMD Multiple Instruction Multiple Data

– Architecture in which each processor has its own instruction and data streams

NUMA Non Uniform Memory Access


Glossary

Object Oriented Programming Programming paradigm in which data and procedures are

encapsulated into objects– objects are defined by their interfaces, i.e., what they do and not

how they do it

– the way the data is represented within an object, and the way the methods which can manipulate it are implemented is hidden from the rest of the program

Network Network topology which allows “perfect shuffle” required for

FFTs to be carried out in parallel– equivalent to Butterfly, Fat tree, and Hypercube


Glossary

OpenMP Message Passing API

– See http://www.openmp.org for details

Packet Size The amount of data that can (has to) be transferred as an atomic

unit– overhead associated with each packet (framing, headers,…)– packet size must be small for a fine-grained machine, otherwise

available bandwidth cannot be used to send useful data

QCD Quantum ChromoDynamics

– theory of the strong interaction, by which strongly interacting elementary particles are built from quarks and gluons

– non-perturbative QCD calculations use Monte Carlo methods on a lattice discretisation of space-time


Glossary

RAM Random Access Memory

RISC Reduced Instruction Set Computer

– arithmetic operations only act on registers

– instructions are of uniform length and duration

• this rule is almost always violated by the inclusion of floating point instructions

– memory access, instruction decoding, and arithmetic can be carried out by separate units in the processor

ROM Read Only Memory


Glossary

SDRAM Synchronous DRAM

– a DRAM chip protocol allowing more overlap of memory access operations

SECDED Single Error Correction Double Error Detection

– the most common form of ECC used for DRAM memories

– usually uses 7 syndrome bits for 32 bit words or 8 syndrome bits for 64 bit words


Glossary

Silicon Compilers Translators from a Hardware Description Language (such as

VHDL) into set of masks from which a (semi) custom chip can be made

SIMD Single Instruction Multiple Data

– architecture in which all processors execute the same instruction at the same time on different data

– instructions are usually broadcast from a single copy of the program

– processors run in lock-step


Glossary

SMP Symmetric MultiProcessor

– Set of processors connected to a shared memory by a common bus or switch

SRAM Static RAM

– bit value stored as state of bistable “flip-flop”

– six transistors required per bit

– used for registers and on-chip caches


Glossary

Static Scheduling Scheduling of instructions (usually by a compiler) to optimise

performance– does not rely on hardware to analyse dynamic behaviour of the

program– may make use of knowledge of “average” behaviour of a program

obtained from profiling

Superscalar architecture having several functional units which can carry

out several operations simultaneously– load/store– integer arithmetic– branch– floating point pipelines


Glossary

Thin Nodes Relatively slow & small processors in a multiprocessor

machine– require fine-grained parallelism


Glossary

Vector Instructions Instructions which carry out the same operation on a whole

stream of data values– reduces memory bandwidth requirements by providing many data

words for one address word

– reduces memory bandwidth requirements by reducing number of instructions (no gain if there is a reasonably large instruction cache)

– requires more register space for temporaries

– some architectures using short vector instructions

• Intel Pentium MMX instructions

• Hitachi PA-RISC extensions


Glossary

VLIW Very Long Instruction Word

– instructions which allow software to control many functional units simultaneously

– hard to program by hand

– allows compilers opportunities for static scheduling

VLSI Very Large Scale Integration

– technology in which a large number of TTL (transistor-transistor logic) circuits an fabricated on a single semiconductor chip

Summary of “Towards Petaflops” Workshop, 24—28 May 1999 A D Kennedy, D C Heggie, S P Booth The...

Documents

Transcript of Summary of “Towards Petaflops” Workshop, 24—28 May 1999 A D Kennedy, D C Heggie, S P Booth The...