HPCA - 7meiyang/ecg700/readings/presentation_spec_benchmarks.pdf · HPCA - 7 Performance...

spec

Standard Performance Evaluation Corporation

Kaivalya M. Dixit

Chief Cook & Bottlewasher, SPECServer Performance, IBM

HPCA - 7

Performance SPECulations - Benchmarks, Friend or Foe

[email protected]

Monterrey, MexicoJanuary 23, 2001

spec


Technology TrendsPopular Benchmarks -- will they ever die?Applications vs. Benchmarks - one perspectiveTop 500 Super Computers Performance Problem Benchmark viewed by: Engr, Mktg, Standards, Boss & UsersSPEC - infrastructureSPEC benchmarksCPU2K MPG - blind submissionCPU2k MPG - one year laterComing attractionsIssuesSummary

Topics

spec


Transistors (1971-2000)Technology (1971-2000)

7173

7577

7981

8385

8789

9193

9597

990.1

1

10

1000

10000

100000

1000000

10000000

100000000

4004

P4

486

P II

P III

mic

ron

s#

of

Tra

ns

isto

rs

spec


Frequency (1971-2000)Technology (1971-2000)

71 73 75 77 79 81 83 85 87 89 91 93 95 97 990.1

1

10

0.1

1

10

100

1000

10000

P III

mic

ron

sM

Hz

4004

486

P4

spec


Chip #Transistors (Million )

Frequency (MHz)

Alpha 15.2 833AMD 37 1200HP 116 552IBM 44 600Sun 29 900SGI 7.2 400

Other Architectures - Today

IBM POWER 4 has announced 170+ million transistors

spec


Tomorrow Copper/silver, Silk, SOI, Electron Beam and other technologies may extend Moore's law for next 10 to 12 yearsSmall Transistors ( 3 atoms )High Clockrates ( 10-20 GHz )System on a chipCustom processorsHigh performance and Low power (oxymoron?)GIPs, Gflops and GDB available to everyoneChallenge Amdahl's Law?

spec


DhrystonesSPECTPCSAP

OracleEEMBCLinpackStreams

Dhrystones

DhampstonesLinpack

PerfectDhampstones

???

Streams

Benchmarks - Yesterday, Today & Tomorrow

More $$$s

Larger

spec


Linpack poor predictor (2 x Linpack = ~10% App. performance)SPECfp95 (2 x SPECfp95 = 2 x App. performance)BW demand on SPECfp2000 higher than App.Icache misses of SPECfp2000 an order of magnitude smaller than App.Mispredicted branches similar to App.Ratio of MemOPs to FPOPS 1-4 with mean of 2 for Apps. but 0.75 to 2 with mean of 1 for BenchmarkWe need to gather and share a lot more data from Apps. Reference: http://www.cs.varginia.edu/~mccalpin/wwc-keynote.html

Applications vs. Benchmarks

spec


Highlights from the Top 100:

ASCI White is new #1 with 4.9 TFlops on LinpackDOE ASCI systems hold the first 4 positions7 systems have Linpack performance above 1 TFlops18 systems have peak performance above 1 TFlops including one commercial system (#15 at Charles Schwab)0.89 Tflops is the entry point for the Top 10Super Computer Rating either MIPS or MFLOPS -- why?

spec


performance = f (MHz,Path Length, CPI,..)Perception: High Performance = High MHzCISC, RISC, VLIW, SMT,CMP, ...ISA, pipe-lining, super scalar, branch predictors, Reg. renaming, OOO, caches, load/store reordering, prefetch, memory, ...Technology, compiler, language, ....SMP, NUMA, MPP, Clusters, ...Benchmarks vs. Applications

Performance Equation

spec


Linpack inspired fma, load / store units, & compiler tricksDhrystone inspired string instructionsStream influenced CPU to memory designsSPEC inspired preprocessors, improved cache, branch predictors, OOO, ISAs, optimizers for graphics, web, compilers, Libs and FDOsTPC drove improvement of OS, database & I/OBoth TPC & SPEC have helped improve H/W and SW scalabilityAll Benchmark Specials are BAD

Benchmark Influence

spec


Performance

BenchmarkEngineer

Workloads

Analysis

Rules

Data

Metrics

Bias

Debate

#1

Lawyer

Marketing

Benchmark Metric = Complex

spec


Performance

BenchmarkMarketing

Papers

Value

Customer

Info

Driver

Bias

#1

Benchmark Metric = Single Number

spec


Performance

BenchmarkStandards

Papers

Rules

More BM

Analysis

Metrics

Bias

Workloads

Benchmark Metric = Full Disclosure

spec


Performance

BenchmarkBoss

Happy

#1

Value

$$$?

Benchmark Metric = Don't care

spec


Performance

BenchmarkUser#1

Value

Benchmark Metric =

spec


Board of DirectorsLegal, HQ, PR

OSG GPC HPG

Committees & People Who Do Real Work!

Infrastructure

60+ Members and Associates

spec


OSG GPC HPGSPECjbb2000 SPECviewperf 6.11 SPEC HPC96 V1.10

SPEC CPU2000 SPECglperf 3.1.2

SPECweb99 SPECapc for Pro/ENGINEER 2000i

SPECjvm98 SPECapc for SolidWorks99

SPECsfs97 SPECapc for Unigraphics V15

Popular Benchmarks

Retired: SPEC89, 92, 95, SPECweb96 ..

0.64

0.490.46

0.83

0.40

Compaq IBM Intel SGI Sun

SP

EC

int2

000/

MH

zSPECint2000/MHz

667

500

0.67

0.87

Gain due to optimization flags

340 733 400 333

11/99

0.62 0.64

0.35

0.82

0.49

Compaq IBM Intel SGI Sun

SP

EC

int2

000/

MH

zSPECint2000/MHz

833

500

0.70

0.88


450 1500 400 900

0.36

0.52

0.65 01/01

One Year Later

0.770.82

0.33

0.90

0.38

Compaq SGI Intel IBM Sun

SP

EC

fp20

00/M

Hz

SPECfp2000 / MHz

0.87Gain due to optimization flags

667 400 733 200 333

0.87

11/99

0.71

0.96

0.35

0.79

0.47

Compaq SGI Intel IBM Sun

SP

EC

fp20

00/M

Hz

SPECfp2000 / MHz


833 400 1500 450 900

1.00

0.79

0.37

0.91

0.54

01/01

One Year Later

spec


Coming Attractions

Released 01/19/2001 - SPECmail2000Server side Java - J2EE Benchmark High Performance Computing - SPEComp2000SPEC Annual Meeting (Jan 24, 2001) will be held in Austin, TXSearching for CPU 200X benchmarksFor latest information for SPEC and TPC go to http://www.spec.org & http://www.tpc.org

spec


IssuesBenchmarking is getting more and more EXPENSIVE & DIFFICULTBenchmark development cycles are getting longer - technology racing aheadBiggest problem with benchmarks is they represent yesterday's workloadsBenchmarks help to MOVE bottlenecks not REMOVE themBenchmarketing takes over benchmarkingOld benchmarks are good members of NRA (Not Relevant Application) club - vigilance necessary

spec


Summary

GIPs, GFLOPs & GDB used for simpler MMIGood Benchmarks will improve H/W & S/WBiggest problem with benchmarks is that often times they represent yesterday's workloadsBenchmarks help to MOVE bottlenecks not REMOVE themBest way to help is provide what you think is a good benchmark to standards organizationSPEC has a bounty program and we are working on future benchmarksQ & A

HPCA - 7meiyang/ecg700/readings/presentation_spec_benchmarks.pdf · HPCA - 7 Performance...

Documents

Transcript of HPCA - 7meiyang/ecg700/readings/presentation_spec_benchmarks.pdf · HPCA - 7 Performance...